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1.  Introduction 


1.1  Purpose 

The  goal  of  this  research  is  to  examine  the  ways  in  which  human  operators  interact  with  unmanned 
ground  vehicles  (UGVs)  and  unmanned  aerial  vehicles  (UAVs).  One  specific  objective  is  to 
evaluate  how  many  robotic  vehicles  an  operator  can  effectively  manage  at  one  time  and  how  the 
operator’s  performance  is  affected  by  the  quality  of  video  image  from  the  robotic  vehicles.  The 
understanding  of  the  robotic  operator’s  span  of  control  is  key  to  successful  operation  of  robotic 
assets  that  are  an  important  part  of  the  U.S.  Army’s  current  force  and  will  be  an  essential  part  in 
the  Army’s  future  force  (Kamsickas,  2003). 

1.2  Background 

The  robotic  operator’s  span  of  control  is  one  of  the  most  important  issues  in  robotic  operational 
environments.  In  recent  years,  research  has  been  conducted  to  investigate  the  robotic  operator’s 
performance  when  more  than  one  unmanned  asset  is  employed,  compared  to  when  only  one  is 
used  (Dixon,  Wickens,  &  Chang,  2003;  Rehfeld,  Jentsch,  Curtis,  &  Fincannon,  2005).  Dixon  et 
al.  examined  pilots’  perfonnance  in  simulated  military  reconnaissance  missions  using  UAV(s). 
They  found  that  pilots  actually  detected  fewer  targets  with  two  UAVs  than  with  a  single  UAV. 
Rehfeld  et  al.  examined  cost  benefits  of  various  human-robot  interaction  (HRI)  teaming  concepts 
by  conducting  a  laboratory  experiment  in  a  scale  military  operations  in  urban  terrain  setting. 

They  compared  one  versus  two  UGVs  and  found  that  the  additional  UGV  did  not  enhance  the 
target  detection  performance  of  the  operator(s).  In  fact,  in  difficult  scenarios,  the  single 
operators  actually  perfonned  worse  with  two  robots  than  with  one  robot. 

Generally  speaking,  the  robotic  operator’s  workload  tends  to  be  higher  when  s/he  has  to 
teleoperate  a  robot  or  manually  intervene  when  the  robot’s  autonomous  operation  encounters 
problems,  compared  to  managing  autonomous  robots  (Dixon  et  al.,  2003;  Schipani,  2003). 

Dixon  et  al.  demonstrated  that  automation  appeared  to  benefit  UAV  pilots’  target  detection 
performance.  Similarly,  Allender  and  Luck  (2005)  reported  that  robotic  operators’  situational 
awareness  (SA)  was  better  when  the  small  UGV  had  a  higher  level  of  automation.  According  to 
Fong,  Thorpe,  and  Baur  (2003),  teleoperation  tends  to  be  challenging  because  operator 
performance  is  “limited  by  the  operator’s  motor  skills  and  his  ability  to  maintain  situational 
awareness...  difficulty  building  mental  models  of  remote  environments...  distance  estimation  and 
obstacle  detection  can  also  be  difficult”  (p.  699). 

In  addition  to  control  modality,  the  communication  channel  between  the  human  operator  and  the 
robot  is  essential  for  effective  perception  of  the  remote  environment.  Factors  such  as  distance, 
obstacles,  or  electronic  jamming  may  pose  challenges  for  maintaining  sufficient  signal  strength 
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(French,  Ghirardelli,  &  Swoboda,  2003).  As  a  result,  the  quality  of  video  “feeds”  that  a 
teleoperator  relies  on  for  remote  perception  may  be  degraded,  and  the  operator’s  performance  in 
distance  and  size  estimation  may  be  compromised  (Van  Erp  &  Padmos,  2003).  The  following 
two  sections  briefly  review  past  research  on  effects  of  slow  frame  rate  (FR)  and  latency  on 
human  perfonnance. 

1.2.1  Frame  Rate 

Common  fonns  of  video  degradation  caused  by  low  bandwidth  include  reduced  FR  (frames  per 
second),  reduced  resolution  of  the  display  (pixels  per  frame),  and  a  lower  gray  scale  (number  of 
levels  of  brightness  or  bits  per  frame)  (Rastogi,  1996).  Piantanida,  Boman,  and  Gille  (1993) 
found  that  participants’  depth  and  egomotion  perception  degraded  when  FRs  dropped.  Similarly, 
Darken,  Kempster,  and  Peterson  (2001)  demonstrated  that  people  had  difficulty  maintaining 
spatial  orientation  in  a  remote  environment  with  a  reduced  bandwidth.  The  participants  also  had 
great  difficulty  in  identifying  objects  in  the  remote  environment.  For  applications  in  virtual 
environments,  many  researchers  recommend  10  Hz  to  be  the  minimum  FR  to  avoid  performance 
degradation  (Watson,  Walker,  Ribarsky,  &  Spaulding,  1998).  Van  Erp  and  Padmos  (2003) 
suggest  that  speed  and  motion  perception  may  be  degraded  if  image  update  rate  is  below  10  Hz. 
Massimino  and  Sheridan  (1994)  demonstrated  that  teleoperation  was  significantly  affected  with  a 
rate  of  five  to  six  frames/second  and  became  almost  impossible  to  perfonn  when  the  FR  dropped 
below  three  frames/second.  According  to  Van  Erp  and  Padmos  (2003),  lowering  the  image 
update  rate  may  affect  speed  estimation  and  braking.  French  et  al.  (2003)  showed  that  reduced 
FRs  (e.g.,  two  or  four  frames/second)  affected  the  teleoperator’s  performance  in  navigation 
duration  (time  to  complete  the  navigation  course)  and  perceived  workload,  ft  was  worth  noting 
that  no  significant  differences  were  found  among  different  FRs  (i.e.,  2,  4,  8,  and  16  fps)  for 
navigation  error,  target  identification  (ID),  and  SA.  The  authors,  however,  recommended  that  no 
fewer  than  eight  frames  per  second  should  be  employed  for  teleoperating  UGVs.  ft  appears  that 
increasing  the  FR  to  higher  than  8  Hz  might  not  greatly  enhance  indirect  driving  performance. 
For  example,  in  a  study  on  teleoperation  of  ground  vehicles,  McGovern  (1991)  did  not  find 
driving  performance  degradation  when  image  update  rates  were  lowered  from  30  to  7.5  Hz. 
According  to  Kolasinski  (1995),  slow  FRs,  which  are  usually  associated  with  visual  lag,  may 
cause  perceived  simulator  sickness.  However,  the  effect  of  slow  FR  on  simulator  sickness  tends 
to  be  indirect  and  can  vary  widely,  based  on  scene  complexity. 

1.2.2  Latency 

Another  video-related  factor  that  might  degrade  the  robotic  operator’s  performance  is  time  delay. 
Time  delay  (i.e.,  latency,  end-to-end  latency,  or  lag)  refers  to  the  delay  between  input  action  and 
(visible)  output  response  and  is  usually  caused  by  the  transmission  of  infonnation  across  a 
communications  network  (MacKenzie  &  Ware,  1993;  Fong  et  al.,  2003).  Studies  of  human 
performance  in  virtual  environments  show  that  people  are  generally  able  to  detect  latency  as  low 
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as  10  to  20  ms  (Ellis,  Mania,  Adelstein,  &  Hill,  2004).  Sheridan  and  Ferrell  (1963)  conducted 
one  of  the  earliest  experiments  on  the  effects  of  time  delay  on  teleoperation.  They  observed  that 
time  delay  had  a  profound  impact  on  the  teleoperator’s  perfonnance,  and  the  resulting  movement 
time  increases  were  well  in  excess  of  the  amount  of  delay.  Based  on  this  and  other  experimental 
results,  Sheridan  (2002)  recommended  that  supervisory  control  and  predictor  displays  be  used  to 
ameliorate  the  negative  impact  of  time  delays  on  teleoperation.  Generally,  when  system  latency 
is  more  than  about  1  second,  operators  begin  to  switch  their  control  strategy  to  “move  and  wait” 
instead  of  continuous  command  to  compensate  for  the  delay  (Lane  et  ah,  2002). 

Research  has  shown  that  time  delays  of  less  than  1  second  can  also  degrade  human  performance 
in  interactive  systems.  In  a  simulated  driving  task,  the  driver’s  vehicle  control  was  found  to  be 
significantly  degraded  by  a  latency  of  170  ms  (Frank,  Casali,  &  Wierville,  1988).  According  to 
Held,  Efstathiou,  and  Greene  (1966),  latency  as  short  as  300  ms  would  make  the  teleoperator 
decouple  his  or  her  commands  from  the  robotic  system’s  response.  Warrick  (as  cited  in  Lane  et 
ah,  2002)  also  showed  that  participants’  compensatory  pursuit  tracking  perfonnance  degraded 
with  a  latency  of  320  ms.  Lane  et  al.  (2002),  on  the  other  hand,  did  not  find  any  performance 
degradation  in  a  three-dimensional  tracking  task  until  the  latency  was  more  than  1  second, 
although  the  authors  reported  that  it  took  the  participants  significantly  longer  to  complete  a 
position  (i.e.,  extraction  and  insertion)  task  when  the  latency  was  more  than  500  ms.  In  a  study 
of  target  acquisition  (TA)  using  the  classic  Fitts’  law  paradigm,  MacKenzie  and  Ware  (1993) 
demonstrated  that  movement  times  increased  by  64%  and  error  rates  increased  by  214%  when 
latency  was  increased  from  8.3  ms  to  225  ms.  A  model  of  modified  Fitts’  law  (with  latency  and 
difficulty  in  having  a  multiplicative  relationship)  was  proposed,  based  on  the  experimental 
results.  In  another  study  of  latency  effects  on  performance  of  grasp  and  placement  tasks,  Watson 
et  al.  (1998)  found  that  when  the  standard  deviation  of  latency  was  above  82  ms,  performance 
degraded  (especially  for  the  placement  task,  which  required  more  frequent  visual  feedback).  It 
was  suggested  that  a  short  variable  lag  could  be  more  detrimental  than  a  longer,  fixed  one  (Lane 
et  al.,  2002).  Over-actuation  (e.g.,  over-steering  and  repeated  command  issuing)  is  also  common 
when  system  delay  is  unpredictable  (Kamsickas,  2003;  Malcolm  &  Lim,  2003).  Additionally, 
time  delay  has  been  associated  with  motion/cyber  sickness,  which  can  be  caused  by  cue  conflict 
(i.e.,  discrepancy  between  visual  and  vestibular  systems)  (Stanney,  Mourant,  &  Kennedy,  1998; 
Kolasinski,  1995). 

1.3  Current  Study 

The  goal  of  this  research  was  to  examine  the  ways  in  which  human  operators  behave  when  they 
are  controlling  robotic  platforms.  The  operator’s  task  was  to  conduct  route  reconnaissance 
missions  in  a  simulated  environment.  During  each  mission,  the  operator  employed  one  or  three 
robots  to  detect  enemy  targets  along  a  designated  route.  Each  participant  conducted  four 
missions,  three  with  a  different  robotic  asset  each,  and  a  final  mission  with  all  three  robotic 
assets  at  their  disposal.  Two  of  the  assets  were  semi-autonomous.  For  these,  operators  assigned 
a  set  of  way-points  and  the  robots  then  traveled  the  route  automatically,  unless  the  operator 


3 


intervened  to  alter  their  behavior.  As  the  robot  traveled,  the  operator  manipulated  the  sensors 
searching  for  targets.  The  semi-autonomous  robots  were  a  UAV  and  a  UGV.  The  third  robot 
was  a  ground  vehicle  requiring  Teleop;  in  other  words,  the  operator  had  to  remotely  drive  this 
vehicle  while  manipulating  its  sensors  to  search  for  targets  at  the  same  time.  All  vehicles  were 
simulated  to  be  equipped  with  camera  sensors,  which  could  be  panned/zoomed  and  could  send 
streaming  video  back  to  the  operator  control  station  (OCS).  As  previously  discussed,  Dixon  et 
al.  (2003)  demonstrated  that  the  pilots’  target  search  perfonnance  improved  when  the  UAV  was 
on  auto-pilot,  compared  to  when  they  had  to  manually  pilot  the  UAV.  In  the  current  study,  target 
detection  for  a  semi-autonomous  versus  manually  piloted  UGV  was  evaluated. 

The  current  study  also  examined  the  issue  of  operator’s  span  of  control  of  robotic  assets.  The 
understanding  of  an  operator’s  span  of  control  is  key  to  successful  employment  of  robotic  assets, 
which  are  increasingly  being  deployed  for  military  operations  (Kamsickas,  2003;  Barnes, 
Cosenzo,  Mitchell,  &  Chen,  2005).  One  of  our  objectives  was  to  examine  the  initial  strategies 
used  when  a  single  operator  is  assigned  the  control  of  multiple  heterogeneous  robotic  vehicles. 
Superficially,  more  assets  should  facilitate  mission  performance  since  operators  will  have  access 
to  different  perspectives  of  the  environment;  however,  the  challenge  of  vehicle  coordination  and 
the  need  to  monitor  multiple  sensor  feeds  might  undermine  the  benefits  of  greater  sensor 
coverage.  In  addition,  control  of  multiple  robotic  vehicles  might  require  additional  training 
beyond  the  training  given  for  the  operation  of  each  individual  vehicle.  Dixon  et  al.  (2003)  and 
Rehfeld  et  al.  (2005)  reported  that  participants  did  not  perform  better  with  two  robots  than  with  a 
single  robot  and  actually  perfonned  worse  in  more  difficult  conditions.  In  the  multiple  asset 
condition  of  the  current  study,  in  contrast  with  the  Dixon  et  al.  and  Rehfeld  et  al.  studies,  we 
used  three  heterogeneous  unmanned  vehicles  (UV)  instead  of  multiple  homogeneous  platforms. 

Another  aim  was  to  investigate  whether  individual  differences  in  spatial  ability  might  impact  the 
performance.  Spatial  ability  is  the  ability  to  navigate  or  manipulate  objects  in  a  two-  or  three- 
dimensional  space  (Eliot,  1984).  Gugerty  (2004)  found  that  UAV  operators  report  difficulty  in 
maintaining  spatial  orientation.  Lathan  and  Tracey  (2002)  showed  that  people  with  higher 
spatial  ability  perfonned  better  in  a  teleoperation  task  through  a  maze.  They  finished  their  tasks 
faster  and  had  fewer  errors. 

Finally,  we  sought  to  investigate  whether  operator  perfonnance  would  be  affected  by  temporal 
aspects  of  the  video  image  transmitted  back  from  the  robotic  vehicles.  In  a  real  situation, 
communication  constraints  might  affect  the  latency  between  robotic  control  input  and  observable 
changes  in  the  sensor  feed  or  might  affect  the  FR  at  which  the  sensor  feed  can  be  displayed 
(Rastogi,  1996).  This  might  have  consequences  for  maintenance  of  SA,  distance  estimation,  and 
target  or  obstacle  detection  (Darken  et  al.,  2001;  Fong  et  al.,  2003;  Van  Erp  &  Padmos,  2003). 

For  one  group  of  participants  (Group  Latency),  a  latency  was  imposed  between  control  input  and 
observable  responses  of  the  Teleop  vehicle.  Such  a  time  delay  is  a  realistic  potential 
consequence  of  the  need  to  transmit  information  between  the  OCS  and  the  robotic  platform.  In 
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our  experiment,  we  employed  a  fixed  latency  of  250  ms,  based  on  the  findings  from  the  literature 
that  latencies  between  225  and  300  ms  would  degrade  human  performance  in  tasks  such  as 
teleoperation,  tracking,  and  TA  (MacKenzie  &  Ware,  1993;  Held  et  ah,  1966;  Warrick,  as  cited 
in  Lane  et  ah,  2002). 

For  the  second  group  (Group  Frame),  no  latency  was  imposed  between  control  input  and 
responses  of  the  Teleop  vehicle;  however,  this  group  had  a  different  manipulation.  For  Group 
Frame,  the  FR  of  the  sensor  video  sent  to  the  OCS  from  all  the  robotic  platforms  decreased  as  a 
function  of  the  distance  between  the  robotic  platform  and  the  OCS.  Consequently,  at  the 
beginning  of  each  mission,  the  FR  was  normal  (i.e.,  25  Hz)  but  decreased  over  the  mission  as  the 
robot  traveled  away  from  the  OCS.  FR  at  the  end  of  a  mission  was  approximately  5  Hz. 

In  order  to  isolate  the  effect  of  the  latency  manipulation,  we  compared  the  Teleop  performance 
data  of  Groups  Latency  and  Frame  from  a  time  period  when  the  FR  for  Group  Frame  was 
nonnal.  This  was  the  first  quarter  of  the  teleop  missions.  In  order  to  isolate  the  effect  of  the  FR 
manipulation,  we  compared  the  performance  results  during  the  first  quarter  (nonnal  FR)  and  the 
last  quarter  (decreased  FR).  The  FR  analysis  included  the  performance  results  of  missions  with 
the  semi-autonomous  platfonns  only  so  as  not  to  contaminate  the  analysis  with  the  effects  of  the 
latency  manipulation  (which  affected  only  the  Teleop  robot).  An  effect  of  FR  should  appear  as  a 
Group  x  Quarter  interaction,  with  the  performance  of  the  two  groups  being  similar  during  the 
first  quarter  but  different  during  the  last  quarter.  The  four  experimental  sessions  are  presented  in 
table  1. 


Table  1.  Type  and  number  of  robotic  assets  and  video  degradation  conditions. 


Robot  Condition 

Video 

Cond. 

Autonomous 

UAV 

Autonomous 

UGV 

Teleop  (UGV) 

Mixed 

Frame 

1  UAV  with 
slow  FR 

1  UGV  with  slow  FR 

1  teleop  with  slow  FR 

1  UAV  with  slow  FR 

1  UGV  with  slow  FR 

1  teleop  with  slow  FR 

Latency 

1  UAV  with 
normal  video 

1  UGV  with  normal 
video 

1  teleop  with  latency 

1  UAV  with  normal  video 

1  UGV  with  normal  video 

1  teleop  with  latency 
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We  were  also  interested  in  examining  if  the  FR  and  latency  manipulations  would  induce  any 
simulator  sickness  symptoms  and  how  gender  differences  would  interact  with  these  factors. 
According  to  the  literature,  slow  FR  and  time  lag  may  lead  to  increased  simulator  sickness,  and 
females  may  be  more  susceptible  (Pausch,  Crea,  &  Conway,  1992).  Simulator  sickness 
susceptibility  also  tends  to  be  a  function  of  the  operator’s  degree  of  control  (Kolasinski,  1995). 

It  was  observed  in  simulation  studies  that  participants  who  generated  input  themselves  tended  to 
report  less  sickness  (Pausch  et  ah,  1992).  In  our  experiment,  although  both  the  UGV  and  the 
Teleop  were  ground  vehicles,  participants  could  anticipate  the  movement  of  the  Teleop  (although 
there  was  a  slight  delay  between  the  input  and  the  movement)  better  than  they  could  with  the 
semi-autonomous  UGV.  The  conflict  between  the  visual  cue  and  the  participants’  own  physical 
state  (basically  stationary)  when  they  controlled  the  UGV  might  result  in  more  severe  sickness. 

It  was  also  reported  in  the  literature  that  altitude  tends  to  be  one  of  the  strongest  contributors  to 
sickness  (Kennedy,  Berbaum,  &  Smith,  1993).  Lower  altitudes  tend  to  induce  more  severe 
sickness  because  of  the  greater  visual  flow  cues  indicating  movement  (Kolasinski,  1995).  In  our 
study,  both  the  UAV  and  the  UGV  were  semi-autonomous  but  one  was  aerial  while  the  other 
was  ground  vehicle.  We  were  interested  in  ascertaining  if  the  UGV  would  induce  more  severe 
sickness  because  of  its  lower  altitude  and  greater  visual  flow. 

In  summary,  the  independent  variables  examined  in  this  current  study  were  number  of  robotic 
assets  (1  versus  3),  type  of  robotic  assets  (UAV,  UGV,  and  Teleop),  and  forms  of  video 
degradation  (slow  frame  rate  and  latency).  It  was  expected  that  operators  would  not  perform 
better  with  three  robots  in  target  detection  tasks.  It  was  also  expected  that  operators  would 
perform  better  with  the  UAV  and  UGV  than  with  the  Teleop.  Both  forms  of  video  degradation 
were  expected  to  affect  operators’  TA  performance.  Participants  with  higher  spatial  ability  were 
expected  to  outperform  those  with  lower  spatial  ability,  in  tenns  of  both  speed  and  accuracy. 


2.  Method 


2,1  Participants 

Thirty  students  (11  females  and  19  males;  27  undergraduates  and  3  graduate  students)  were 
recruited  from  the  University  of  Central  Florida  and  participated  in  the  study.  The  ages  of  the 
participants  ranged  from  18  to  33  (female:  M=  21,  SD  =  4.07;  male:  M=  19,  SD  =  1.9).  Of  the 
30  participants,  25  self  reported  being  at  least  good  with  computers,  4  reported  as  excellent,  and 
1  expert.  As  for  video  game  experiences,  27  participants  reported  playing  at  least  some  video 
games.  Participants  were  paid  $50  or  given  class  credit  for  their  participation  in  the  experiment. 
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2.2  Apparatus 


2.2.1  Simulator 


The  experiment  was  conducted  with  the  Embedded  Combined  Arms  Team  Training  and  Mission 
Rehearsal  (ECATT-MR)  test  bed  at  the  Simulation  and  Training  Technology  Center  of  Research, 
Engineering,  and  Development  Command  (RDECOM)  in  Orlando,  Florida.  The  operator  control 
display  for  the  ECATT-MR  is  illustrated  in  figure  1.  The  test  bed  was  equipped  with  a  steering 
wheel  and  gas  and  break  pedals  for  control  of  the  teleoperated  vehicle.  Mechanical  buttons  on 
the  steering  device  provided  for  control  of  the  targeting  and  weapons  systems  of  the  teleoperated 
vehicle  (weapons  were  not  used  in  this  study,  however).  The  OneSAF  (Semi-Automated  Forces) 
test  bed  was  used  to  provide  the  simulated  environments  and  the  computer-generated  forces. 


Friendly  asset  camera 

view  (UAV/UGV)  View  from  Teleop  Teleop  Turret  -  targeting 


UAV/UGV  Controls  Situational  Awareness  Map  Teleop  Status  Info 


Figure  1.  User  interface  of  ECATT-MR  test  bed. 


2.2. 1.1  Tele-operating  the  Robotic  Vehicle 

Driving  the  Teleop  vehicle  was  similar  to  driving  a  car,  although  the  operator  had  to  first  select 
the  “drive”  function  on  the  touch  screen  and  then  start  driving  using  the  pedal  and  the  steering 
yoke,  which  also  had  several  buttons  for  controlling  the  weapons  and  targeting  system  of  the 
robotic  asset  (figure  2).  When  a  target  appeared  in  the  robotic  vehicle’s  field  of  view,  it  also 
appeared  as  a  string  of  letters  and  numbers  on  the  “target  list”  (each  target’s  ID  was  unique)  on 
the  lower  right  portion  of  the  Teleop  status  display.  Once  a  target  had  been  located,  the  operator 
first  drove  the  Teleop  within  range,  and  then  s/he  could  rotate  the  main  gun  360  degrees  and 
raise  and  lower  it  to  adjust  the  turret  view  (with  a  crosshair  in  the  center)  by  controlling  the 
steering  yoke.  The  right  palm  grip,  however,  needed  to  be  depressed  in  order  for  the  operator  to 
rotate  or  raise/lower  the  gun.  Once  the  target  was  inside  the  gun’s  crosshairs,  the  operator  could 
press  the  “lase”  button  (on  the  right  handle  of  the  steering  console)  to  determine  the  range  of  the 
unit,  which  was  shown  on  the  Teleop  turret  display  (top  right  screen). 
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Figure  2.  Diagram  of  yoke  control  buttons  (from  unpublished  ECATT-MR  manual  prepared  by 
RDECOM  STTC,  2004). 


2.2. 1.2  Controlling  the  Semi-autonomous  Robotic  Vehicles 

The  operator  used  the  UV  status  panel  for  managing  the  UAV  and  the  UGV  (figure  3).  The 
“lase”  function  was  under  “engagement”  and  the  operator  could  tilt  and  pan  the  camera  sensor 
for  each  asset  by  selecting  the  “sensor  view”  button.  The  operator  selected  the  “assign  task” 
button,  for  example,  to  move  an  asset  or  order  the  UAV  to  hover.  Typically,  at  the  start  of  a 
scenario,  the  operator  placed  way  points  on  the  SA  map  (lower  center  screen)  using  its  Point 
Editor  and  then  “assign  task”  to  send  the  robot  into  its  reconnaissance  mission.  Both  the  UAV 
and  the  UGV  traveled  at  20  kph,  and  the  default  altitude  for  the  UAV  was  100  m.  When  the 
operator  detected  a  target,  s/he  first  halted  the  robot  and  adjusted  the  sensor  view  by  pressing  the 
appropriate  buttons  (e.g.,  left,  right,  down,  up,  etc.)  so  the  crosshair  was  on  the  target  before 
firing  the  laser  at  the  target  by  pressing  the  “lase”  button.  A  detailed  SA  map  is  presented  in 
figure  4. 
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Figure  3.  UV  status  display  -  sensor  view. 


Figure  4.  SA  map  display  (MD). 
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2.2.2  Questionnaires 

The  Cube  Comparison  Test  (Educational  Testing  Service,  2005)  was  administered  to  assess 
participants’  spatial  ability.  The  Cube  Comparison  Test  requires  participants  to  compare,  in 
3  minutes,  2 1  pairs  of  six-sided  cubes  and  determine  if  the  rotated  cubes  are  the  same  or 
different. 

Appendix  A  presents  the  demographics  questionnaire  administered  at  the  beginning  of  the 
training  session. 

Perceived  workload  was  measured  by  the  National  Aeronautics  and  Space  Administration’s  task 
load  index  (NASA-TLX)  questionnaire  (appendix  B).  The  NASA-TLX  is  a  self-reported 
questionnaire  of  perceived  demands  in  nine  areas:  mental,  physical,  temporal,  effort  (mental  and 
physical),  frustration,  performance,  visual,  cognitive,  and  psychomotor  (Hart  &  Staveland, 

1988).  Participants  were  asked  to  evaluate  their  perceived  workload  level  in  these  areas  on  10- 
point  scales. 

The  simulator  Sickness  Questionnaire  (appendix  C)  was  used  to  evaluate  participants’  simulator 
sickness  symptoms  (Kennedy,  Lane,  Berbaum,  &  Lilienthal,  1993).  The  Simulator  Sickness 
Questionnaire  consists  of  a  checklist  of  16  symptoms.  Each  symptom  is  related  in  terms  of 
degrees  of  severity  (none,  slight,  moderate,  severe).  A  Total  Severity  (TS)  score  can  be  derived 
by  a  weighted  scoring  procedure  and  reflects  overall  discomfort  level  (Kolasinski,  1995). 

A  usability  questionnaire  (appendix  D)  was  constructed,  based  on  the  one  used  in  the  Unmanned 
Combat  Demonstration  (UCD)  study,  since  the  test  bed  used  in  our  study  was  modeled  after  the 
crew  station  investigated  in  the  UCD  study  (Kamsickas,  2003).  Specifically,  the  questionnaire 
included  the  following  sections:  MD,  reporting  (RPT),  UV  control  and  status,  teleoperation,  TA, 
crew  station  display  and  screens,  yoke  and  pedal  assembly  (YPA),  and  other  equipment. 
Participants  indicated  their  level  of  agreement  with  the  items  using  7-point  numerical  scales 
(strongly  disagree  [1],  disagree  [2],  somewhat  disagree  [3],  neutral  [4],  somewhat  agree  [5], 
agree  [6],  and  strongly  agree  [7]).  Participants  were  also  given  an  opportunity  to  provide 
comments  to  support  or  clarify  their  numeric  responses.  The  comments,  in  addition  to  the 
numeric  responses,  provided  the  researchers  with  further  insight  as  to  the  participants’  opinions 
about  the  crew  station. 

Finally,  a  strategy  questionnaire  was  constructed  to  gain  further  insights  into  participants’ 
favorite  strategies  for  using  the  robotic  assets  (appendix  E). 

2.3  Procedure 

Fifteen  participants  were  randomly  assigned  to  either  the  Latency  or  FR  group.  Each  participant 
conducted  four  missions,  three  with  a  different  robotic  asset  each,  and  a  final  mission  with  all 
three  robotic  assets,  as  illustrated  in  table  1 .  The  order  of  presentation  of  the  single-robot 
conditions  was  counterbalanced,  while  the  three-robot  (mixed)  condition  was  always  the  last. 
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Thus,  participants  had  a  chance  to  complete  a  mission  with  each  asset  singly  before  conducting  a 
mission  with  all  three. 

Participants  received  training  and  practice  in  the  tasks  they  would  need  to  conduct  during  an 
initial  session  that  took  approximately  3  hours  (see  appendix  A).  Participants  returned  one  week 
later  to  complete  the  experiment.  Before  the  experimental  session,  participants  took  the  Cube 
Comparison  Test  (Educational  Testing  Service,  2005),  the  scores  of  which  were  later  used  to 
designate  a  participant’s  spatial  ability.  After  the  Cube  Comparison  Test,  participants  were 
given  some  refresher  practice  and  then  were  asked  to  complete  four  route-reconnaissance 
missions.  For  each  mission,  they  were  given  a  specific  route  to  travel  with  the  requirement  to 
detect  and  fire  a  laser  at  as  many  targets  as  they  could  find  and  to  reach  the  end  point  within  30 
minutes.  Each  mission  occurred  across  the  same  terrain  map  but  used  a  different  route  and 
direction  of  travel.  Assignment  of  specific  routes  to  asset  conditions  was  counterbalanced  across 
participants.  Each  route  was  approximately  4  km  and  consisted  of  an  assembly  area,  a  starting 
point,  two  checkpoints,  and  an  end  point.  Participants  were  instructed  to  issue  a  location  report 
at  each  of  these  spots.  Each  mission  allowed  for  the  detection  of  12  targets,  which  were  a 
mixture  of  enemy  vehicles  and  dismounted  Soldiers.  Upon  detection  of  a  target,  participants 
were  to  send  a  contact  report  and  fire  a  laser  at  the  target.  Periodically,  the  warning  signal  for 
“Communications  Fault”  illuminated  and  the  participants  needed  to  double  click  the  button  to 
reset  it. 

The  workload  questionnaire  (NASA-TLX)  and  the  Simulator  Sickness  Questionnaire  were  given 
at  the  end  of  each  scenario  to  assess  the  participants’  perceived  workload  as  well  as  simulator 
sickness  symptoms.  Upon  completion  of  the  experimental  session,  the  usability  and  the  strategy 
questionnaires  were  given. 

In  addition  to  the  questionnaire  data,  mission  performance  data  (such  as  number  of  laser  firings, 
number  of  targets  fired  upon  with  a  laser,  time  to  complete  missions,  etc.)  were  automatically 
captured  by  the  software. 


3.  Results 


3.1  Task  Completion  Time 

The  proportion  of  participants  who  finished  the  mission  in  the  allotted  time  (30  minutes)  was 
significantly  lower  in  the  mixed  asset  condition,  compared  with  any  of  the  single-asset  conditions, 
Cochran’s  Q  (3  df)  =  31.93,/?  <  .001.  The  mean  percent  of  participants  completing  each  mission 
was  at  least  89%  for  all  the  single-asset  conditions  but  was  only  44.8%  for  the  mixed  asset 
condition.  Time  to  complete  each  mission  was  also  affected  by  asset  condition,  F( 3,  51)  =  21.18, 
p  <.001.  Tukey  HSD  (Honestly  Significant  Difference)  tests  showed  that  participants  took 
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significantly  less  time  to  complete  the  Teleop  condition  (M=  17.9  min)  than  any  of  the  other 
three  conditions.  Among  these  three  conditions,  participants  took  significantly  less  time  to 
complete  the  UAV  condition  (M=  23.1  min)  than  the  mixed  condition  (M=  27.1  min).  Time  to 
complete  the  UGV  mission  was  intermediate  to  these  (M=  24.3  min).  Note  that  participants 
could  not  control  the  speeds  of  the  UAV  or  UGV,  although  they  could  hover  the  UAV  or  halt  the 
UGV. 


3.2  Target  Detection  and  Acquisition 

Table  2  lists  several  measures  relating  to  target  detection  and  acquisition.  When  participants 
detected  a  target,  they  were  instructed  to  fire  a  laser  at  it;  however,  sometimes  they  fired  at  the 
same  target  more  than  once.  Other  times,  they  fired  at  their  own  assets  by  mistake  (friendly), 
and  yet  other  times,  they  fired  at  nothing  (missed).  These  data  were  analyzed  with  spatial  ability 
scores  as  a  covariate.  A  mixed  analysis  of  covariance  (ANCOVA)  with  one  within-subject 
factor  (Asset)  and  one  between-subject  factor  (video  degradation)  and  spatial  ability  as  the 
covariate  was  performed.  The  analysis  revealed  that  asset  condition  (UAV,  UGV,  Teleop,  or 
Mixed)  significantly  affected  number  of  targets  fired  upon,  F( 3,  75)  =  3.305,  p  <  .05,  with 
Teleop  being  the  lowest  (see  figure  5).  Since  the  Teleop  condition  was  also  completed  the  most 
quickly,  we  examined  targets  detected  per  minute.  This  was  also  significantly  lower  in  the 
Teleop  condition,  F( 2,  50)  =  24.80,/?  <  .001,  indicating  that  the  fewer  targets  fired  upon  in  this 
condition  were  not  simply  attributable  to  the  condition’s  shorter  duration.  Friendly  fires  (i.e., 
firing  at  the  Teleop)  were  also  evaluated.  However,  the  differences  were  not  significant. 


Table  2.  Target  detection  and  acquisition  performance  (means  and  standard  deviations). 


Measures 

UAV 

UGV 

Teleop 

Mixed 

Number  of  targets  fired  upon  (Max  =  12) 

10.33 

(2.02) 

9.34 

(1.4) 

5 

(2.03) 

9.24 

(2.28) 

Targets  detected  per  minute* 

0.46 

0.40 

0.27 

0.35 

Total  number  of  fires 

14.63 

(4.93) 

12.1 

(2.61) 

8.27 

(5.69) 

14.66 

(5.81) 

Number  of  contact  reports 

12.17 

(4.39) 

9.66 

(2.11) 

8.6 

(3.83) 

12.07 

(3.84) 

Number  of  missed  fires 

1.8 

(2.4) 

3 

(3.3) 

8.97 

(7.9) 

3.97 

(3.41) 

Firing  at  friendly  vehicles 

.97 

(1.79) 

.24 

(.64) 

NA 

.38 

(.68) 

*Computed  only  for  participants  who  reached  the  end  of  the  route  within  30  minutes. 


The  detection  data  for  the  mixed  scenario  were  segregated  by  asset  type,  and  a  comparison  with 
the  single-asset  missions  is  shown  in  figure  6.  There  was  a  significant  interaction  between  asset 
and  number  of  assets,  F( 2,  54)  =  22.90,/?  <  .001.  In  the  mixed  condition,  UAV  was  used  to  fire 
a  laser  at  about  the  same  number  of  targets  as  the  UAV-alone  condition.  In  contrast,  the  UGV 
and  the  Teleop  were  used  much  less  frequently  in  the  mixed  condition  than  when  they  were  the 
sole  asset. 
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Figure  5.  Detection  performance. 
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Figure  6.  Targets  fired  upon  in  single  scenarios  versus  mixed  scenario.  (Error 
bars  depict  standard  error  of  the  mean.) 

There  was  no  significant  effect  of  latency  or  FR  on  the  target  detection  data  when  the  data  were 
analyzed  as  outlined  in  the  introduction.  The  effect  of  latency  on  the  detection  data  was  in  the 
expected  direction  (poorer  firing  performance  in  Group  Latency  than  Group  Frame  during 
quarter  1)  but  failed  to  be  reliable.  It  is  possible  that  restricting  the  analysis  to  only  the  first 
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quarter  of  the  Teleop  missions  limited  our  power  to  detect  an  effect,  since  there  were  only  three 
targets  to  detect. 

To  evaluate  the  effect  of  frame  rate,  hit  rates  (number  of  targets  fired  upon  divided  by  total  firing 
attempts)  for  UAV  and  UGV  in  the  first  and  last  quarters  were  compared.  For  the  UGV,  hit  rate 
was  lower  in  the  fourth  than  the  first  quarter,  and  this  decrease  was  greater  in  Group  Frame  than 
Group  Latency.  For  the  UAV,  hit  rate  was  essentially  flat  across  the  quarters.  The  predicted 
Group  x  Quarter  interaction  failed  to  be  significant  when  both  the  UGV  and  UAV  were 
considered  (the  Group  x  Quarter  x  Asset  interaction  was  not  significant). 

3.3  Perceived  Workload 

A  mixed  analysis  of  variance  (ANOVA)  with  one  within-subject  factor  (Asset)  and  two  between- 
subject  factors  (video  condition  and  gender)  demonstrated  that  participants’  self-assessment  of 
workload  was  significantly  affected  by  Asset  condition,  F( 3,  54)  =  6.437,/?  <  .005.  The  perceived 
workload  was  higher  in  the  mixed  condition  (M=  72.3)  than  the  single-asset  conditions 
(M’s  =  60.9,  61.0,  64.6  for  Teleop,  UAV,  and  UGV  conditions,  respectively).  There  was  a 
moderately  significant  interaction  for  asset,  video  condition,  and  gender,  F( 3,  54)  =  2.628, 
p  =  .059.  The  ratings  are  graphically  presented  in  figure  7. 


Figure  7.  Perceived  workload. 


3.4  Strategies  for  Handling  Multiple  Vehicles 

At  the  end  of  the  mixed  asset  condition,  participants  were  asked  to  characterize  how  they 
managed  the  multiple  robots  by  selecting  items  from  the  list  of  choices  shown  in  table  3.  More 
than  75%  of  the  participants  indicated  that  their  main  strategy  was  to  send  the  UAV  out  in  front. 
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The  experimenters  observed  that  people  tended  to  search  the  route  with  the  platform  they  felt 
most  comfortable  with  (typically  the  UAV)  and  then  simply  steered  the  remaining  platforms  to 
the  end  point  without  doing  much  additional  searching.  Thus,  there  was  little  evidence  that 
people  attempted  to  coordinate  the  movement  of  the  platfonns  or  to  use  them  to  their  best 
advantage.  There  were  only  two  “other”  responses  that  implied  any  deeper  thought  about  how  to 
use  multiple  assets.  These  were  “If  target  appeared,  I  used  the  most  appropriate  platform  (one 
with  the  best  view)”  and  “used  the  UAV  to  find  Soldiers  and  the  UGV  to  find  tanks.” 


Table  3.  Strategies  for  managing  multiple  robotic  assets  during  the  mixed  scenario. 


Options 

Percent  of  partici¬ 
pants  who  selected 
as  a  method 

Percent  of  participants 
who  selected  as  the 
most  dominant  method 

1 .  Move  one  robot  from  start  to  finish,  and  then  do  the  next 
one,  then  the  last. 

10 

0 

2.  Keep  UAV  in  front  of  the  ground  robots. 

90 

75.8 

3.  Keep  Teleop  in  front  of  the  other  robots. 

3 

0 

4.  Keep  UGV  in  front  of  the  other  robots. 

3 

0 

5.  Keep  all  robots  no  more  than  one  checkpoint  apart. 

7 

10.3 

6.  If  a  target  appeared  on  the  map,  move  the  closest  robot 
to  view  it. 

27 

3.4 

7.  If  a  target  appeared  on  the  map,  drive  the  Teleop  into 
position  to  view  it. 

4 

3.4 

8.  If  a  target  appeared  on  the  map,  fly  the  UAV  to  the 
nearest  checkpoint  to  view  it. 

43 

3.4 

9.  Other 

27 

3.4 

3.5  Simulator  Sickness 

The  TS  score  was  computed  for  each  participant  (see  appendix  F  for  the  scoring  procedure). 
Participants  rated  their  simulator  sickness  as  the  most  severe  in  the  Mixed  condition  (M=  20.82) 
and  the  least  severe  in  the  Teleop  condition  (M=  17.67).  The  mean  TS  scores  for  UAV  and 
UGV  are  18.08  and  19.95,  respectively.  A  mixed  ANOVA  with  one  within-subject  factor 
(Asset)  and  two  between-subjects  factors  (video  condition  and  gender)  was  performed.  None  of 
the  main  effects  were  significant.  Generally,  a  TS  score  of  more  than  20  is  considered  moderate. 
It  appears  that  our  participants  experienced  more  severe  simulator  sickness  when  they  operated 
all  three  robots  or  when  they  operated  the  UGV  (whose  mean  TS  score  approached  20).  Our 
simulation  rarely  produced  severe  symptoms. 

3.6  Usability  Questionnaire 

Twenty-nine  participants  completed  the  usability  questionnaire,  in  which  participants  rated  the 
survey  items  on  7-point  scales,  with  7  being  “strongly  agree”  and  1  being  “strongly  disagree.” 

To  evaluate  the  difference  in  usability  ratings  between  the  two  experimental  groups,  t-tests  were 
conducted.  Generally,  Latency  group  participants  rated  their  experiences  as  less  ideal  than  did 
the  FR  group.  The  Latency  group  participants  felt  the  pedals  were  more  sensitive  to  control 
Teleop’s  movement  than  did  the  FR  group;  they  felt  the  video  imagery  for  TA  from  the  Teleop 
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was  less  clear;  the  number  of  displays  was  less  appropriate  to  accomplish  the  required  tasks;  they 
felt  “lost”  when  working  with  the  crew  station  more  often  (table  4).  Participants  from  both  FR 
and  Latency  groups  observed  the  sensitivity  of  the  yoke  and  the  difficulty  of  maneuvering  it. 
Some  participants  from  the  Latency  group  commented  that  it  was  easy  to  veer  off  the  road  when 
driving  fast.  General  results  of  the  usability  questionnaire  and  some  selected  comments  from  the 
participants  are  available  in  appendix  G. 


Table  4.  Usability  survey  results. 


Statement 

Group  Latency 

Group  FR 

P  value 

The  pedals  were  too  sensitive  to  control  Teleop’s 
movement. 

M=  3.33 
(SD=  1.46) 

M=  1.84 
(SD  =  1.99) 

p  =  .035 

The  video  imagery  for  TA  from  the  Teleop  is 
clear. 

M=  4.58 
(SD  =  1.44) 

M  —  6.3 
(SD  =  .82) 

p  =  .003 

There  are  not  enough  displays  to  accomplish  the 
required  tasks. 

M=  2.64 
(SD  =  1.34) 

M=  1.45 
(SD  =  .69) 

© 

II 

I  sometimes  feel  “lost”  when  working  with  the 
crew  station. 

M=  2.96 
(SD  =  1.74) 

M=  1.73 
(SD  =  .79) 

p  =  .039 

3.7  Spatial  Ability 

Participants’  spatial  ability  was  identified  by  their  Cube  Comparison  Test  scores.  Spatial  ability 
scores  were  found  to  be  positively  correlated  with  the  number  of  targets  fired  upon  in  the  UAV, 
UGV,  and  the  mixed  conditions,  r  =  .457,  .362,  and  .502  respectively,  all  p's  <  .05.  The 
correlations  between  spatial  ability  and  mission  completion  times  were  all  negative  (i.e., 
participants  with  higher  spatial  ability  completed  their  missions  faster);  however,  only  the 
correlation  in  the  UAV  condition  was  significant,  r=  -.372,  p  <  .05.  The  correlation  in  the 
Teleop  condition  was  marginally  significant,  r=  -.301  ,p  =  .056. 

3.8  Gender  Differences 

There  were  no  significant  gender  differences  in  any  of  our  performance  measures,  except  for  the 
moderately  significant  difference  in  the  number  of  contact  reports  sent  by  males  (M  =  9.847)  and 
females  (M=  1 1.675),  F(l,25)  =  3.96, p  =  .058.  Additionally,  for  firing  at  friendly  vehicles, 
there  was  a  significant  interaction  between  asset  and  gender,  F{ 3,  23)  =  3.027, p  =  .05.  Females 
fired  more  at  friendly  vehicles  in  the  mixed  condition,  while  males  fired  more  at  friendly 
vehicles  with  the  UAV.  Gender  differences  in  perceived  workload  and  simulator  sickness  have 
been  presented  in  previous  sections. 


4.  Discussion 


These  findings  suggest  that  giving  robotic  operators  additional  assets  may  not  be  beneficial  for 
enhancing  target  detection  performance.  Essentially,  participants  failed  to  discover  more  targets 
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with  three  robots,  compared  with  the  UAV  alone  or  the  UGV  alone.  Moreover,  participants 
experienced  a  higher  workload  in  the  mixed  asset  condition,  and  more  than  half  of  participants 
failed  to  complete  it  in  the  allotted  time.  It  appears  that  giving  multiple  assets  to  robotic 
operators  may  be  counterproductive.  Participants  exhibited  little  natural  tendency  to  coordinate 
the  use  of  assets. 

These  findings  are  consistent  with  those  of  other  robotic  control  studies  (Dixon  et  ah,  2003; 
Rehfeld  et  ah,  2005).  In  Dixon  et  ah,  pilots  detected  fewer  targets  with  two  UAVs  than  with  a 
single  UAV.  Rehfeld  et  al.  found  that  giving  a  second  UGV  to  a  single  operator  or  a  two-person 
team  failed  to  enhance  the  individual’s  or  team’s  target  detection  perfonnance.  In  fact,  in 
difficult  scenarios,  the  single  operators  actually  perfonned  worse  with  two  robots  than  with  one. 
On  the  other  hand,  the  two-person  teams  perfonned  more  than  twice  as  well  as  the  one-person 
condition  in  those  difficult  scenarios,  regardless  of  how  many  assets  were  used.  These  findings 
echoed  what  have  been  observed  in  the  field  (e.g.,  using  robots  for  search  and  rescue  efforts  in 
Murphy,  2004)  that  remote  perception  is  still  one  of  the  most  fundamental  challenges  for  robotic 
operators.  The  findings  of  Dixon  et  al.,  Rehfeld  et  al.,  and  the  current  study  suggest  that, 
regardless  of  the  types  and  homogeneity  of  the  robotic  platforms,  additional  assets  do  not  appear 
to  be  beneficial  for  reconnaissance  types  of  tasks,  at  least  when  information  must  be  gleaned 
from  streaming  video. 

Participants  did  not  appear  to  take  advantage  of  the  multiple  perspectives  available  in  the  mixed 
asset  condition.  This  might  be  for  several  reasons.  First,  three  robots  appeared  to  be  more  than 
the  operators  could  handle.  Second,  participants  were  not  specifically  given  instructions  about 
how  to  coordinate  multiple  assets.  Finally,  the  existing  user  interface  did  not  support  effective 
integration  of  sensor  infonnation  from  multiple  platforms.  An  improved  user  interface  should 
benefit  the  operator’s  integration  of  information  from  different  sources.  With  the  development 
of  the  Future  Combat  System  (FCS),  one  can  realistically  expect  situations  when  operators  need 
to  integrate  information  from  more  than  one  platform,  potentially  from  aerial  and  ground 
sources.  According  to  the  literature,  exocentric  frame  of  reference,  which  the  UAV  used,  is 
more  suitable  for  global  awareness  perfonnance  and  search  tasks  than  egocentric  frame  of 
reference,  which  the  UGV  used  (McCormick,  Wickens,  Banks,  &  Yeh,  1998;  Wang,  2004).  If 
an  operator  has  to  use  a  ground  robotic  vehicle  for  tasks  involving  search,  some  additional 
information  from  the  exocentric  perspective  might  be  beneficial  for  enhancing  the  search 
performance.  Salzman,  Dede,  Loftin,  and  Ash  (1998)  showed  that  a  combination  of  egocentric 
and  exocentric  frames  of  references  had  benefits  for  visualization  of  complex  information. 
However,  displays  for  integrating  information  from  different  frames  of  references  (e.g., 
exocentric  and  egocentric)  present  potential  human  perfonnance  issues  that  need  to  be  carefully 
evaluated  (Thomas  &  Wickens,  2000).  Research  has  shown  that  integrating  infonnation  across 
egocentric  and  exocentric  views  can  be  challenging  for  the  operator  (Olmos,  Wickens,  &  Chudy, 
2000).  In  addition,  operators  may  be  susceptible  to  saliency  effects  and  anchoring  heuristic/bias. 
Salient  infonnation  on  one  display  may  catch  most  of  the  operator’s  attention,  and  the  operator 
may  form  an  inaccurate  judgment  because  information  from  the  other  sources  is  not  properly 
attended  to  and  integrated.  More  research  in  effective  ways  to  present  data  from  multiple 
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sources  and  perspectives  will  provide  very  useful  information  to  the  FCS  design  efforts. 

Research  programs  such  as  the  U.S.  Army’s  HRI  Army  Technology  Objective  have  started  to 
explore  advanced  user  interface  design  concepts  and  innovative  technologies  to  enhance  robotic 
operator  performance  (Barnes  et  ah,  2005). 

Results  from  our  study  suggest  that  the  detection  performance  in  the  Teleop  condition  may  have 
been  negatively  affected  by  the  participants’  driving  (i.e.,  Teleop)  task.  It  seems  somewhat 
counterintuitive  that  participants  completed  the  Teleop  mission  more  quickly  than  the  UAV  or 
UGV  missions.  Components  of  the  missions  included  maneuvering,  searching  the  sensor  image, 
firing  lasers,  and  RPT.  For  the  semi-autonomous  assets,  comparatively  little  maneuvering  was 
required.  It  seems  that  for  the  Teleop  condition,  a  focus  on  maneuvering  was  at  the  expense  of 
time  spent  searching.  These  findings  are  consistent  with  Allender  and  Luck  (2005)  and  Dixon  et 
al.  (2003)  that  robotic  operators  demonstrated  higher  SA  when  the  robot’s  level  of  automation 
was  higher.  Allender  and  Luck  (2005)  suggested  that  the  attention  on  (manual)  robotic  control 
might  have  distracted  the  operators  from  focusing  on  the  vehicle’s  location,  which  was  the 
study’s  measure  of  SA.  In  Dixon  et  al.,  pilots  found  more  targets  when  their  UAV(s)  were 
autonomous  than  when  they  were  teleoperated. 

We  were  somewhat  surprised  how  little  the  two  types  of  display  degradation  affected  operator 
performance.  The  responses  to  the  usability  questionnaire,  on  the  other  hand,  provided  more 
insight  about  the  differences  between  the  latency  and  FR  conditions.  For  instance,  the  Latency 
group  agreed  more  strongly  than  the  FR  group  that  the  pedals  were  too  sensitive  to  control  the 
Teleop.  Given  the  initial  time  delay  between  their  input  actions  and  the  output  responses, 
participants  might  have  over-compensated  and  therefore  felt  that  the  pedals  were  “too  sensitive.” 
Overall,  however,  it  did  not  appear  that  a  slow  FR  at  about  5  Hz  and  a  latency  of  250  ms  had  a 
significant  impact  on  our  participants’  robotic  control  and  reconnaissance  perfonnance. 

Our  results  of  superior  performance  by  participants  with  higher  spatial  ability  using  the  robotic 
assets  are  consistent  with  past  research  (e.g.,  Lathan  &  Tracey,  2002;  Vincow,  1998). 

Participants  with  higher  spatial  ability  were  found  to  perform  better  in  both  speed  and  accuracy 
across  platforms.  Their  superior  perfonnance  was  especially  consistent  when  they  used  the 
UAV.  Our  findings  support  the  recommendation  by  Lathan  and  Tracey  (2002)  that  military 
missions  can  benefit  from  the  selection  of  personnel  with  higher  spatial  ability  to  operate  robotic 
devices.  Also,  training  interventions  that  could  enhance  the  spatial  interpretations  required  might 
be  of  benefit  (Rodes,  Brooks,  &  Gugerty,  2005). 


5.  Future  Directions 


In  the  ensuing  study,  we  are  going  to  evaluate  if  gunners  in  an  FCS  vehicle  such  as  the  Mounted 
Combat  System  (MCS)  are  able  to  effectively  detect  targets  in  their  immediate  environment 
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while  operating  robotic  assets  in  a  remote  environment.  According  to  Mitchell  (2005),  which 
used  the  Improved  Performance  Research  Integration  Tool  modeling  to  examine  workload  for 
MCS  crew  members,  the  gunner  is  the  only  viable  option  for  controlling  robotic  assets, 
compared  to  the  other  two  positions  (i.e.,  commander  and  driver).  In  this  ensuing  study,  we  are 
going  to  simulate  the  MCS  environment  and  perform  an  experiment  to  further  examine  the 
workload  and  performance  of  the  combined  position  of  gunner  and  robotic  operator.  Past 
research  in  dual  task  performance  suggests  that  operators  may  encounter  difficulties  when  both 
tasks  involve  focal  vision  (Horrey  &  Wickens,  2004).  Horrey  and  Wickens  (2004)  demonstrated 
that  participants  could  not  effectively  detect  road  hazards  while  operating  in-vehicle  devices. 
Because  of  the  heavy  visual  load  for  both  the  monitoring  (i.e.,  gunner’s)  task  and  the  robotic 
control  tasks,  we  expect  the  gunner/robotic  operator  will  have  difficulties  in  perfonning  both 
tasks  effectively. 
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Appendix  A.  Demographic  Questionnaire 


Participant# _  Age _  Major _  Date _  Gender _ 

1.  What  is  the  highest  level  of  education  you  have  had? 

Less  than  4  yrs  of  college _  Completed  4  yrs  of  college _  Other _ 

2.  When  did  you  use  computers  in  your  education?  ( Circle  all  that  apply ) 

Grade  School  Jr.  High  High  School 

Technical  School  College  Did  Not  Use 

3.  Where  do  you  currently  use  a  computer?  (Circle  all  that  apply 1 

Home  Work  Library  Other _  Do  Not  Use 

4.  For  each  of  the  following  questions,  circle  the  response  that  best  describes  you. 

How  often  do  you: 

Use  a  mouse?  Daily,  Weekly,  Monthly,  Once  every  few  months,  Rarely,  Never 

Use  a  joystick?  Daily,  Weekly,  Monthly,  Once  every  few  months.  Rarely,  Never 

Use  a  touch  screen?  Daily,  Weekly,  Monthly,  Once  every  few  months,  Rarely,  Never 

Use  icon-based  programs/software? 

Daily,  Weekly,  Monthly,  Once  every  few  months.  Rarely,  Never 
Use  programs/software  with  pull-down  menus? 

Daily,  Weekly,  Monthly,  Once  every  few  months.  Rarely,  Never 
Use  graphics/drawing  features  in  software  packages? 

Daily,  Weekly,  Monthly,  Once  every  few  months.  Rarely,  Never 
Use  E-mail?  Daily,  Weekly,  Monthly,  Once  every  few  months,  Rarely,  Never 

Operate  a  radio  controlled  vehicle  (car,  boat,  or  plane)? 

Daily,  Weekly,  Monthly,  Once  every  few  months.  Rarely,  Never 

Play  computer/video  games? 

Daily,  Weekly,  Monthly,  Once  every  few  months.  Rarely,  Never 

5.  Which  type(s)  of  computer/video  games  do  you  most  often  play  if  you  play  at  least  once  every  few  months? 

6.  Which  of  the  following  best  describes  your  expertise  with  computer?  (check  V  one) 

_ Novice 

_ Good  with  one  type  of  software  package  (such  as  word  processing  or  slides) 

_ Good  with  several  software  packages 

_ Can  program  in  one  language  and  use  several  software  packages 

_ Can  program  in  several  languages  and  use  several  software  packages 

7.  Are  you  in  your  usual  state  of  health  physically?  YES  NO 
If  NO,  please  briefly  explain: 

8.  How  many  hours  of  sleep  did  you  get  last  night? _ hours 

9.  Do  you  have  normal  color  vision?  YES  NO 

10.  Do  you  have  prior  military  service?  YES  NO  If  Yes,  how  long _ 


25 


Intentionally  left  blank 


26 


Appendix  B.  NASA-TLX  Questionnaire 


Please  rate  your  overall  impression  of  demands  imposed  on  you  during  the  exercise. 

1.  Mental  Demand:  How  much  mental  and  perceptual  activity  was  required  (e.g.,  thinking,  looking,  searching, 
etc.)?  Was  the  task  easy  or  demanding,  simple  or  complex,  exacting  or  forgiving? 

LOW  H-l-H-H-l-l-l  HIGH 
123456789  10 

2.  Physical  Demand:  How  much  physical  activity  was  required  (e.g.,  pushing,  pulling,  turning,  controlling, 
activating,  etc.)?  Was  the  task  easy  or  demanding,  slow  or  brisk,  slack  or  strenuous,  restful  or  laborious? 

LOW  H-l-l-l-l-l-l-l-l  HIGH 
123456789  10 

3.  Temporal  Demand:  How  much  time  pressure  did  you  feel  due  to  the  rate  or  pace  at  which  the  task  or  task 
elements  occurred?  Was  the  pace  slow  and  leisurely  or  rapid  and  frantic? 

LOW  H-l-l-l-l-l-l-l-l  high 
123456789  10 

4.  Level  of  Effort:  How  hard  did  you  have  to  work  (mentally  and  physically)  to  accomplish  your  level  of 
performance? 


LOW  H-l-l-l-l-l-l-l-l  high 
123456789  10 

5.  Level  of  Frustration:  How  insecure,  discouraged,  irritated,  stressed  and  annoyed  versus  secure,  gratified,  content, 
relaxed  and  complacent  did  you  feel  during  the  task? 

LOW  H-l-l-l-l-l-l-l-l  HIGH 
123456789  10 

6.  Performance:  How  successful  do  you  think  you  were  in  accomplishing  the  goals  of  the  task  set  by  the 
experimenter  (or  yourself)?  How  satisfied  were  you  with  your  performance  in  accomplishing  these  goals? 

LOW  l-l-l-l-l-l-l-l-l-l  HIGH 
123456789  10 

7.  Please  mark  the  indicated  loading  that  most  closely  matches  the  work  performed  by  your  visual,  cognitive,  and 
motor  efforts  on  the  task  just  completed. 

Visual  LOW  Load  |— |— |— |— |— |— |— |— |— |  HIGH  Load 

123456789  10 

Cognitive  LOW  Load  |— |— |— |— |— |— |— |— |— |  HIGH  Load 

123456789  10 

Psvchomotor  (Relating  to  the  physical  activities  associated  with  mental  processes 

LOW  Load  |— |— |— |— |— |— |— |— |— |  HIGH  Load 
123456789  10 
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Appendix  C.  Simulator  Sickness  (Current  Health  Status)  Questionnaire 


ID _  Time  &  Date 

Instructions:  Please  indicate  how  you  feel  right  now  in  the  following  areas,  by  circling  the 
word  that  applies. 


1. 

General  Discomfort 

None 

Slight 

Moderate 

Severe 

2. 

Fatigue 

None 

Slight 

Moderate 

Severe 

3. 

Headache 

None 

Slight 

Moderate 

Severe 

4. 

Eye  Strain 

None 

Slight 

Moderate 

Severe 

5. 

Difficulty  Focusing 

None 

Slight 

Moderate 

Severe 

6. 

Increased  Salivation 

None 

Slight 

Moderate 

Severe 

7. 

Sweating 

None 

Slight 

Moderate 

Severe 

8. 

Nausea 

None 

Slight 

Moderate 

Severe 

9. 

Difficulty  Concentrating 

None 

Slight 

Moderate 

Severe 

10. 

Fullness  of  Head 

None 

Slight 

Moderate 

Severe 

11. 

Blurred  vision 

None 

Slight 

Moderate 

Severe 

12. 

Dizzy  (Eyes  Open) 

None 

Slight 

Moderate 

Severe 

13. 

Dizzy  (Eyes  Closed) 

None 

Slight 

Moderate 

Severe 

14. 

Vertigo* 

None 

Slight 

Moderate 

Severe 

15. 

Stomach  Awareness** 

None 

Slight 

Moderate 

Severe 

16. 

Burping 

None 

Slight 

Moderate 

Severe 

Vertigo  is  a  disordered  state  in  which  the  person  or  his/her  surroundings  seem  to  whirl  dizzily:  giddiness. 
Stomach  awareness  is  usually  used  to  indicate  a  feeling  of  discomfort  which  is  just  short  of  nausea. 

Are  there  any  other  symptoms  you  are  experiencing  right  now?  If  so,  please  describe  the  symptom(s)  and 
rate  its/their  severity  below.  Use  the  other  side  if  necessary. 
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Appendix  D.  Usability  Survey 


Instructions 


This  questionnaire  is  designed  to  elicit  your  reactions  for  the  use  of  the  crewstation.  Your 
responses  will  help  us  understand  what  aspects  of  the  crewstation  you  find  inefficient  or  difficult 
to  use.  This  questionnaire  provides  an  opportunity  to  evaluate  the  usability  of  the  system.  We 
will  use  your  responses  to  influence  the  design  of  the  future  crewstation.  To  as  great  a  degree  as 
possible,  think  about  all  the  tasks  that  you  have  perfonned  with  the  crewstation  while  you  answer 
the  questions. 

Read  each  statement  or  question  and  indicate  your  response  to  the  statement  or  question  by 
circling  the  value  you  feel  best  gauges  your  answer  to  the  question  on  the  scale  underneath  it.  If 
a  statement  does  not  apply,  circle  N/A.  Use  the  space  on  the  right  to  provide  any  comments  that 
would  help  us  improve  the  design  of  the  crewstation  interface.  Comments  clarifying  answers  are 
extremely  useful. 

Map  Display 

1.  I  used  the  Mapping  display  often.  Comments 

Strongly  DISAGREE  | — | — | — | — | — |— - 1  Strongly  AGREE  N/A 

1  2  3  4  5  6  7 

2.  The  button  labels  accurately  represented  the  buttons'  functionality. 

Strongly  DISAGREE  | — | — | — | — | — | — |  Strongly  AGREE  N/A 

1  2  3  4  5  6  7 


3.  The  information  presented  on  the  mapping  display  was  easy  to  read. 

Strongly  DISAGREE  | — | — | — |— -|— -|— -|  Strongly  AGREE  N/A 

1  2  3  4  5  6  7 


4.  The  map  resolution  made  it  difficult  for  me  to  select  a  specific  location. 

Strongly  DISAGREE  | — | — | — | — | — | — |  Strongly  AGREE  N/A 
1  2  3  4  5  6  7 

5.  It  was  easy  to  understand  what  the  icons  used  on  the  map  displays 
represented. 

Strongly  DISAGREE  | — Strongly  AGREE  N/A 
1  2  3  4  5  6  7 


6.  I  was  able  to  control  map  graphics  effectively. 

Strongly  DISAGREE  | — | — | — |— -| — | — |  Strongly  AGREE  N/A 

1  2  3  4  5  6  7 
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7.  Icons  representing  different  ARVs  were  easy  to  distinguish. 

Strongly  DISAGREE  | — | — | — | — | — |— -|  Strongly  AGREE  N/A 
1  2  3  4  5  6  7 


8.  The  icons  that  represent  ARVs  are  too  large. 

Strongly  DISAGREE  | — | — | — | — | — | — |  Strongly  AGREE  N/A 

1  2  3  4  5  6  7 


9.  Geographic  orientation  of  the  map  terrain  features  makes  sense. 

Strongly  DISAGREE  | — | — | — | — | — | — |  Strongly  AGREE  N/A 

1  2  3  4  5  6  7 


10.  The  map  terrain  features  (roads,  water,  vegetation)  were  difficult  to  see. 

Strongly  DISAGREE  | — | — | — |— - 1 — |  Strongly  AGREE  N/A 

1  2  3  4  5  6  7 

11.  The  resolution  of  the  location  grid  lines  was  sufficient  to  allow  me  to 
accomplish  my  goals. 

Strongly  DISAGREE  |— - 1— -| — |— -|— -|— -|  Strongly  AGREE  N/A 

1  2  3  4  5  6  7 


12.  The  symbols  that  were  used  did  not  make  sense. 

Strongly  DISAGREE  | — | — | — |— -|— -|— -|  Strongly  AGREE  N/A 

1  2  3  4  5  6  7 


13.  The  orientation  of  the  ARVs  is  represented  clearly. 

Strongly  DISAGREE  | — | — | — | — | — | — |  Strongly  AGREE  N/A 

1  2  3  4  5  6  7 


14.  The  icons  that  represent  non-ARV  units  are  too  small. 

Strongly  DISAGREE  | — | — | — | — | — | — |  Strongly  AGREE  N/A 
1  2  3  4  5  6  7 

15.  I  would  like  a  way  to  customize  the  background  colors  on  the  map. 

Strongly  DISAGREE  | — | — | — | — | — | — |  Strongly  AGREE  N/A 
1  2  3  4  5  6  7 


16.  The  map  was  my  main  source  of  situation  awareness. 

Strongly  DISAGREE  | — | — | — |— -| — |— - 1  Strongly  AGREE  N/A 

1  2  3  4  5  6  7 


17.  Text  on  the  map  status  display  was  easy  to  understand. 

Strongly  DISAGREE  | — | — | — |— -|— -|— -|  Strongly  AGREE  N/A 
1  2  3  4  5  6  7 

18.  It  would  have  been  helpful  to  have  more  information  presented  on  the  map 
display. 

Strongly  DISAGREE  | — | — | — |— - 1 — |  Strongly  AGREE  N/A 
1  2  3  4  5  6  7 


19.  I  used  the  cursor  center  function  often. 

Strongly  DISAGREE  | — | — | — | — | — | — |  Strongly  AGREE  N/A 

1  2  3  4  5  6  7 


Comments 
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20.  Overall  the  mapping  display  was  easy  to  use. 

Strongly  DISAGREE  | — | — | — |— -|— -| — |  Strongly  AGREE  N/A 

1  2  3  4  5  6  7 


Reporting 

1.  The  button  labels  accurately  represented  the  buttons'  functionality. 

Strongly  DISAGREE  | — | — | — | — | — | — |  Strongly  AGREE  N/A 
1  2  3  4  5  6  7 


2.  Much  of  the  information  presented  on  the  display  was  NOT  helpful. 

Strongly  DISAGREE  | — | — | — | — | — | — |  Strongly  AGREE  N/A 
1  2  3  4  5  6  7 


3.  I  sometimes  felt  'lost'  navigating  the  reporting  display  screens. 

Strongly  DISAGREE  | — | — | — | — | — | — |  Strongly  AGREE  N/A 

1  2  3  4  5  6  7 


4.  Composing  a  SITREP  was  an  easy  task  to  accomplish. 

Strongly  DISAGREE  | — |— -| — |— -| — | — |  Strongly  AGREE  N/A 
1  2  3  4  5  6  7 


5.  Composing  a  SITREP  took  too  long  to  accomplish. 

Strongly  DISAGREE  | — | — | — | — | — | — |  Strongly  AGREE  N/A 

1  2  3  4  5  6  7 

6.  The  entry  fields  for  composing  a  SITREP  were  found  in  the  order  I  expected 
them  to  be. 

Strongly  DISAGREE  | — | — | — | — | — | — |  Strongly  AGREE  N/A 

1  2  3  4  5  6  7 


7.  There  was  not  enough  information  displayed  on  the  reporting  screen. 
Strongly  DISAGREE  | — | — | — | — | — | — |  Strongly  AGREE  N/A 
1  2  3  4  5  6  7 


8.  Composing  a  SPOT  report  was  an  easy  task  to  accomplish. 

Strongly  DISAGREE  | — | — | — | — | — | — |  Strongly  AGREE  N/A 

1  2  3  4  5  6  7 


9.  Composing  a  SPOT  report  took  too  long  to  accomplish. 

Strongly  DISAGREE  | — | — | — | — | — | — |  Strongly  AGREE  N/A 

1  2  3  4  5  6  7 

10.  The  entry  fields  for  composing  a  SPOT  report  were  found  in  the  order  I 
expected  them  to  be. 

Strongly  DISAGREE  | — | — | — | — | — | — |  Strongly  AGREE  N/A 

1  2  3  4  5  6  7 


11.  Using  the  reporting  display  to  compose  reports  was  an  easy  process  overall. 

Strongly  DISAGREE  | — | — | — | — | — | — |  Strongly  AGREE  N/A 

1  2  3  4  5  6  7 


Comments 


Comments 


33 


Unmanned  Vehicle  Control  &  Status 

1.  I  used  the  unmanned  vehicle  (UV)  status  display  often. 

Strongly  DISAGREE  | — | — | — | — | — | — |  Strongly  AGREE  N/A 

1  2  3  4  5  6  7 


2.  There  was  adequate  information  displayed  with  regard  to  the  state  of  the 
UVs. 

Strongly  DISAGREE  | — | — | — | — | — | — |  Strongly  AGREE  N/A 
1  2  3  4  5  6  7 


3.  The  text  on  the  UV  detailed  information  screen  was  easy  to  read. 

Strongly  DISAGREE  | — |— -| — | — | — | — |  Strongly  AGREE  N/A 
1  2  3  4  5  6  7 


4.  The  information  presented  in  the  display  was  easy  to  interpret. 

Strongly  DISAGREE  | — | — | — | — | — | — |  Strongly  AGREE  N/A 

1  2  3  4  5  6  7 

5.  The  colors  used  to  indicate  the  status  of  various  UV  resources  (ammo,  fuel, 
etc.)  made  sense. 

Strongly  DISAGREE  | — | — | — | — | — | — |  Strongly  AGREE  N/A 
1  2  3  4  5  6  7 


6.  Requesting  control  of  an  UV  was  easy  to  do. 

Strongly  DISAGREE  | — | — | — | — | — | — |  Strongly  AGREE  N/A 

1  2  3  4  5  6  7 


7.  The  button  labels  accurately  represented  the  buttons'  functionality. 

Strongly  DISAGREE  | — | — | — | — | — | — |  Strongly  AGREE  N/A 

1  2  3  4  5  6  7 

8.  The  icons  used  to  depict  UV  resources  (fuel,  communication  status,  ammo, 
etc.)  and  the  state  of  the  resources  was  easy  to  interpret. 

Strongly  DISAGREE  | — |— -| — | — | — | — |  Strongly  AGREE  N/A 

1  2  3  4  5  6  7 


9.  The  UV  control  display  was  easy  to  use. 

Strongly  DISAGREE  | — | — | — | — | — | — |  Strongly  AGREE  N/A 
1  2  3  4  5  6  7 


Comments 


Comments 
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Teleoperation 


1.  I  used  the  teleoperation  display  often. 

Strongly  DISAGREE  | — | — | — | — | — | — |  Strongly  AGREE  N/A 
1  2  3  4  5  6  7 


2.  The  yoke  made  it  difficult  to  control  the  ARVs. 

Strongly  DISAGREE  | — — | — | — |  Strongly  AGREE  N/A 

1  2  3  4  5  6  7 


3.  The  pedals  were  too  sensitive  to  control  ARV  movement. 

Strongly  DISAGREE  | — | — | — | — | — | — |  Strongly  AGREE  N/A 

1  2  3  4  5  6  7 


4.  The  print/  text  presented  on  the  teleoperation  display  was  easy  to  read. 
Strongly  DISAGREE  | — | — | — | — | — | — |  Strongly  AGREE  N/A 
1  2  3  4  5  6  7 


5.  The  yoke  was  too  sensitive  to  control  ARV  direction. 

Strongly  DISAGREE  | — | — | — | — | — | — |  Strongly  AGREE  N/A 
1  2  3  4  5  6  7 


6.  The  video  update  rate  was  sufficient  to  teleoperate  the  ARV  at  high  speed. 

Strongly  DISAGREE  | — | — |— -| — | — | — |  Strongly  AGREE  N/A 
1  2  3  4  5  6  7 


7.  The  video  update  rate  was  sufficient  to  teleoperate  the  ARV  at  low  speed. 

Strongly  DISAGREE  | — | — | — | — | — | — |  Strongly  AGREE  N/A 
1  2  3  4  5  6  7 


8.  The  streaming  video  was  sufficient  to  allow  me  to  teleoperate  the  ARV. 

Strongly  DISAGREE  | — | — | — | — | — | — |  Strongly  AGREE  N/A 
1  2  3  4  5  6  7 

9.  The  teleoperation  display  did  not  provide  enough  information  (speed,  direction, 
etc.)  on  ARV  status. 

Strongly  DISAGREE  | — | — | — | — | — | — |  Strongly  AGREE  N/A 
1  2  3  4  5  6  7 


10.  I  used  the  map  display  while  teleoperating  an  ARV  to  help  me  maintain  the 
location  of  an  ARV. 

Strongly  DISAGREE  | — | — | — | — | — | — |  Strongly  AGREE  N/A 

1  2  3  4  5  6  7 


11.  There  was  a  significant  time  delay  that  made  controlling  the  ARV  difficult. 

Strongly  DISAGREE  | — — | — | — |  Strongly  AGREE  N/A 
1  2  3  4  5  6  7 

12.  The  Teleoperation  display  provided  too  much  information  (speed,  direction,  etc.) 
on  ARV  status. 

Strongly  DISAGREE  | — | — | — | — | — | — |  Strongly  AGREE  N/A 
1  2  3  4  5  6  7 


Comments 


Comments 


35 


13.  The  backup  feature  was  very  useful  when  an  ARV  got  stuck. 

Strongly  DISAGREE  | — | — | — | — | — | — |  Strongly  AGREE  N/A 

1  2  3  4  5  6  7 

14.  I  had  to  teleoperate  the  ARV  often. 

Strongly  DISAGREE  | — | — | — | — | — | — |  Strongly  AGREE  N/A 

1  2  3  4  5  6  7 

15.  The  button  labels  accurately  represented  the  buttons'  functionality. 

Strongly  DISAGREE  | — | — | — | — | — | — |  Strongly  AGREE  N/A 

1  2  3  4  5  6  7 

16.  I  found  the  Teleoperation  display  easy  to  use  to  control  ARVs. 

Strongly  DISAGREE  | — | — | — | — | — | — |  Strongly  AGREE  N/A 

1  2  3  4  5  6  7 

Target  Acquisition 

1.  I  used  the  Target  Acquisition  (TA)  display  often. 

Strongly  DISAGREE  | — | — |— -| — | — | — |  Strongly  AGREE  N/A 

1  2  3  4  5  6  7 

2.  The  print/text  used  on  the  TA  display  was  easy  to  read. 

Strongly  DISAGREE  | — | — | — | — | — | — |  Strongly  AGREE  N/A 

1  2  3  4  5  6  7 

3.  The  layout  of  the  TA  screen  made  sense. 

Strongly  DISAGREE  | — | — | — | — | — | — |  Strongly  AGREE  N/A 

1  2  3  4  5  6  7 

4.  The  video  imagery  for  TA  from  the  ARVs  is  clear. 

Strongly  DISAGREE  | — | — | — | — | — | — |  Strongly  AGREE  N/A 

1  2  3  4  5  6  7 

5.  The  information  presented  on  the  TA  screen  is  not  helpful. 

Strongly  DISAGREE  | — | — | — | — |— -| — |  Strongly  AGREE  N/A 

1  2  3  4  5  6  7 

6.  The  TA  display  should  contain  additional  information. 

Strongly  DISAGREE  | — | — | — | — | — | — |  Strongly  AGREE  N/A 

1  2  3  4  5  6  7 

7.  The  orientation  of  the  icons  (ARV  gun  direction)  presented  on  the  display 

made  sense. 

Strongly  DISAGREE  | — | — | — | — | — | — |  Strongly  AGREE  N/A 

1  2  3  4  5  6  7 

8.  The  aiming  reticule  was  easy  to  use. 

Strongly  DISAGREE  | — | — | — | — | — | — |  Strongly  AGREE  N/A 

1  2  3  4  5  6  7 

9.  The  button  labels  accurately  represented  their  intended  functions. 

Strongly  DISAGREE  | — | — | — | — | — | — |  Strongly  AGREE  N/A 

1  2  3  4  5  6  7 
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10.  It  was  easy  to  select  an  ARV  to  engage  a  target. 

Strongly  DISAGREE  | — | — | — | — | — | — |  Strongly  AGREE  N/A 
1  2  3  4  5  6  7 

11.  I  would  like  to  see  more  functionality  added  to  the  Target  Acquisition 
display. 

Strongly  DISAGREE  | — | — | — | — | — | — |  Strongly  AGREE  N/A 
1  2  3  4  5  6  7 


12.  There  is  not  enough  information  presented  on  the  TA  screen. 

Strongly  DISAGREE  | — | — | — | — | — |— -|  Strongly  AGREE  N/A 
1  2  3  4  5  6  7 


13.  This  display  made  it  easy  to  engage  targets. 

Strongly  DISAGREE  | — |— -| — | — | — | — |  Strongly  AGREE  N/A 
1  2  3  4  5  6  7 


General  Usability 

I.  Crewstation  Display  and  Screens. 

Definitions: 

•  A  screen  is  defined  as  the  physical  object  that  may  be  handled. 

•  A  display  is  the  software  image  that  is  displayed  on  the  screen. 

•  A  hard  button  is  a  physical  button  on  the  track  ball. 

•  A  soft  button  is  a  computer-generated  image  of  a  button  displayed  on  the  screen 


1.  The  touchscreen  buttons  were  large  enough  to  use  effectively.  Comments 

Strongly  DISAGREE  | — | — | — | — | — | — |  Strongly  AGREE  N/A 

1  2  3  4  5  6  7 

2.  The  print/  text  on  the  crewstation  soft  buttons  was  clear,  and  easy  to 
read. 

Strongly  DISAGREE  | — | — | — | — | — | — |  Strongly  AGREE  N/A 

1  2  3  4  5  6  7 

3.  The  point  at  which  my  finger  touches  is  not  identical  to  the  point 
indicated  by  the  touchscreen  (there  are  parallax  problems). 

Strongly  DISAGREE  | — | — | — | — | — | — |  Strongly  AGREE  N/A 

1  2  3  4  5  6  7 


4.  The  layout  of  the  hard  buttons  made  sense. 

Strongly  DISAGREE  | — |— -| — | — | — | — |  Strongly  AGREE  N/A 

1  2  3  4  5  6  7 
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5.  The  layout  of  the  hard  buttons  made  sense. 

Strongly  DISAGREE  | — |— -|— -|— -| — | — |  Strongly  AGREE  N/A 
1  2  3  4  5  6  7 


6.  It  is  hard  to  select  small  items  when  using  the  touchscreen. 

Strongly  DISAGREE  | — | — | — | — | — | — |  Strongly  AGREE  N/A 
1  2  3  4  5  6  7 


7.  The  number  of  displays  is  appropriate  to  accomplish  the  required  tasks. 

Strongly  DISAGREE  | — |— -| — | — | — | — |  Strongly  AGREE  N/A 

1  2  3  4  5  6  7 


8.  I  had  trouble  selecting  the  appropriate  display  to  use. 

Strongly  DISAGREE  | — |— -| — |— -| — | — |  Strongly  AGREE  N/A 

1  2  3  4  5  6  7 

9.  I  frequently  made  errors  in  selecting  the  correct  display  to  use  when 
attempting  to  accomplish  a  task. 

Strongly  DISAGREE  | — |— -| — — |  Strongly  AGREE  N/A 
1  2  3  4  5  6  7 


10.  There  are  not  enough  displays  to  accomplish  the  required  tasks. 

Strongly  DISAGREE  | — | — | — | — | — | — |  Strongly  AGREE  N/A 

1  2  3  4  5  6  7 
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11.  I  preferred  using  the  touchscreen  to  the  trackball  for  interacting  with  the 
displays. 

Strongly  DISAGREE  | — | — | — — |  Strongly  AGREE  N/A 
1  2  3  4  5  6  7 


12.  It  was  easy  to  learn  to  use  the  basic  features  of  the  crewstation. 

Strongly  DISAGREE  | — |— -|— -|—  -| — | — |  Strongly  AGREE  N/A 

1  2  3  4  5  6  7 

13.  Frequently,  inadvertent  contact  with  the  touchscreen  causes  functions  to 
activate  unintentionally. 

Strongly  DISAGREE  | — |— -|— -|— -| — | — |  Strongly  AGREE  N/A 

1  2  3  4  5  6  7 

14.  Throughout  the  crewstation,  the  same  terminology  is  used  to  indicate  the 
same  information. 

Strongly  DISAGREE  | — |— -| — — |  Strongly  AGREE  N/A 
1  2  3  4  5  6  7 


15.  I  sometimes  feel  "lost"  when  working  with  the  crewstation. 

Strongly  DISAGREE  | — | — | — | — |—  -| — |  Strongly  AGREE  N/A 

1  2  3  4  5  6  7 


16.  I  had  a  hard  time  finding  information  I  needed  on  the  crewstation. 

Strongly  DISAGREE  | — |— -| — | — | — | — |  Strongly  AGREE  N/A 
1  2  3  4  5  6  7 

17.  The  crewstation  presents  system  messages  in  a  consistent  format  and 
location. 

Strongly  DISAGREE  | — |— -| — | — | — | — |  Strongly  AGREE  N/A 
1  2  3  4  5  6  7 

18.  It  was  easy  to  understand  what  the  icons  used  on  the  crewstation 
displays  represented. 

Strongly  DISAGREE  | — |— - 1 — | — | — |  Strongly  AGREE  N/A 
1  2  3  4  5  6  7 


19.  The  crewstation  responds  too  slowly. 

Strongly  DISAGREE  | — |— -|— -|— -| — | — |  Strongly  AGREE  N/A 

1  2  3  4  5  6  7 


20.  The  crewstation  keeps  me  informed  about  what  it  is  doing. 

Strongly  DISAGREE  | — | — | — | — | — | — |  Strongly  AGREE  N/A 

1  2  3  4  5  6  7 

21.  The  messages  that  appear  on  the  crewstation  display  are  hard  to 
understand. 

Strongly  DISAGREE  | — | — | — | — | — | — |  Strongly  AGREE  N/A 

1  2  3  4  5  6  7 
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22.  The  crewstation  gives  appropriate  warning  messages  when  I  am  about  to 
make  a  serious  mistake. 

Strongly  DISAGREE  | — | — | — | — Strongly  AGREE  N/A 
1  2  3  4  5  6  7 


23.  With  the  crewstation,  if  I  make  a  mistake  I  can  correct  it  easily. 

Strongly  DISAGREE  | — |— -| — |— -| — | — |  Strongly  AGREE  N/A 

1  2  3  4  5  6  7 

24.  It  is  obvious  which  command  button  brings  up  the  crewstation  display  I 
need  next. 

Strongly  DISAGREE  | — |— -|— -|— -| — | — |  Strongly  AGREE  N/A 
1  2  3  4  5  6  7 

25.  It  was  difficult  to  select  a  specific  control  on  the  crewstation  touch 
screen. 

Strongly  DISAGREE  | — — |— -| — |  Strongly  AGREE  N/A 
1  2  3  4  5  6  7 


26.  Information  is  appropriately  arranged  on  the  crewstation  displays. 
Strongly  DISAGREE  | — | — | — | — | — | — |  Strongly  AGREE  N/A 
1  2  3  4  5  6  7 


27.  Color-coding  on  the  crewstation  displays  is  helpful. 

Strongly  DISAGREE  | — | — |— - 1 — | — | — |  Strongly  AGREE  N/A 

1  2  3  4  5  6  7 


28.  The  symbols  that  were  used  were  hard  to  learn. 

Strongly  DISAGREE  | — |— -| — |— -| — |— -|  Strongly  AGREE  N/A 

1  2  3  4  5  6  7 


29.  Overall,  I  think  the  crewstation  was  easy  to  learn. 

Strongly  DISAGREE  | — |— -| — |— -| — | — |  Strongly  AGREE  N/A 

1  2  3  4  5  6  7 


30.  Overall,  I  think  the  crewstation  was  easy  to  work  with. 

Strongly  DISAGREE  | — | — | — | — | — | — |  Strongly  AGREE  N/A 

1  2  3  4  5  6  7 


II.  Yoke  and  Pedal  Assembly. 

1.  The  buttons  on  the  yoke  were  easy  to  use. 

Strongly  DISAGREE  | — | — | — | — | — | — |  Strongly  AGREE  N/A 
1  2  3  4  5  6  7 


2.  There  were  too  many  buttons  on  the  yoke  to  choose  from. 

Strongly  DISAGREE  | — |—  -|—  -| — | — | — |  Strongly  AGREE  N/A 
1  2  3  4  5  6  7 


3.  The  bump  cursor  was  easier  to  use  than  the  buttons  on  the  screen. 

Strongly  DISAGREE  | — |—  -| — | — | — | — |  Strongly  AGREE  N/A 

1  2  3  4  5  6  7 
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4. 


Comments 


I  used  the  bump  cursor  to  navigate  the  buttons  on  the  displays  frequently. 

Strongly  DISAGREE  | — |— -| — |— -| — | — |  Strongly  AGREE  N/A 
1  2  3  4  5  6  7 


5.  The  yoke  was  too  sensitive  to  over-steer. 

Strongly  DISAGREE  | — | — | — | — | — | — |  Strongly  AGREE  N/A 

1  2  3  4  5  6  7 


6.  It  was  easy  to  slew  too  the  target  using  the  yoke. 

Strongly  DISAGREE  | — | — | — | — | — | — |  Strongly  AGREE  N/A 

1  2  3  4  5  6  7 


7.  It  was  easy  to  maintain  crosshairs  on  the  target  with  the  yoke. 

Strongly  DISAGREE  | — |— -| — |— -| — | — |  Strongly  AGREE  N/A 
1  2  3  4  5  6  7 


ITT.  Other  Equipment. 

1.  I  used  the  keyboard  often  throughout  the  scenario. 

Strongly  DISAGREE  | — |—  -|—  -| — | — | — |  Strongly  AGREE  N/A 

1  2  3  4  5  6  7 


2.  I  used  the  trackball  often  throughout  the  scenario. 

Strongly  DISAGREE  | — |—  -|—  -| — | — | — |  Strongly  AGREE  N/A 

1  2  3  4  5  6  7 


3.  The  keyboard  was  in  a  position  that  was  easily  accessible. 

Strongly  DISAGREE  | — | — | — | — | — | — |  Strongly  AGREE  N/A 

1  2  3  4  5  6  7 

4.  It  was  easy  to  adjust  the  positioning  of  hardware  (seat,  yoke  placement, 

screens,  etc.)  in  the  crewstation. 

Strongly  DISAGREE  | — | — | — | — | — | — |  Strongly  AGREE  N/A 

1  2  3  4  5  6  7 
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Intentionally  left  blank 
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Appendix  E.  Strategy  Questionnaire 


Which  of  the  following  describes  the  method  you  used  on  the  last  scenario,  when  you  had  all 
three  robots  to  control?  (Check  all  that  apply) 

□  1 .  Move  one  robot  from  start  to  finish,  and  then  do  the  next  one,  then  the  last. 

□  2.  Keep  the  UAV  in  front  of  the  ground  robots. 

□  3.  Keep  the  Teleop  in  front  of  the  other  robots. 

□  4.  Keep  the  UGV  in  front  of  the  other  robots. 

□  5.  Keep  all  robots  no  more  than  one  checkpoint  apart. 

□  6.  If  a  target  appeared  on  the  map,  move  the  closest  robot  to  view  it. 

□  7.  If  a  target  appeared  on  the  map,  drive  the  Teleop  into  position  to  view  it. 

□  8.  If  a  target  appeared  on  the  map,  fly  the  UAV  to  the  nearest  checkpoint  to  view  it. 

□  9.  Other  (describe) 


Of  the  answers  above  that  you  checked,  please  rank  order  the  strategies  from  most  important  to 
least  important. 

Please  estimate  the  number  of  targets  you  acquired  with  each  vehicle. 

UAV _ i 

UGV _ , 

Teleop _ . 
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Intentionally  left  blank 
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Appendix  F.  Scoring  Procedure  for  the  Simulator  Sickness  Questionnaire 


Symptoms  scored  0  (None)  -  3  (Severe) 

Nausea  -  Sum  of  General  discomfort,  increased  salivation,  sweating,  nausea,  diff 
concentrating,  stomach  awareness,  burping 

Oculomotor  -  Sum  of  general  discomfort,  fatigue,  headache,  eye  strain,  diff  focusing,  diff 
concentrating,  blurred  vision 

Disorientation  -  Sum  of  diff  focusing,  nausea,  fullness  of  head,  blurred  vision,  dizzy 
(eyes  open),  dizzy  (eyes  closed),  vertigo 

Total  Severity  Score  =  (Nausea  +  Oculomotor  +  Disorientation)  x  3.74 
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Intentionally  left  blank 
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Appendix  G.  General  Results  of  the  Usability  Questionnaire  and  Selected 
Comments  from  Participants 


I.  General  Results  of  the  Usability  Questionnaire. 

The  following  sections  address  other  general  results  of  the  usability  questionnaire,  focusing  on 
items  in  which  the  modal  response  reflected  the  opinions  of  nearly  half  of  the  respondents. 

Map  Display  (MD).  Of  the  20  MD  items,  there  were  3  in  which  almost  half  of  the  respondents 
were  in  agreement.  The  statement  for  MD  item  number  2  (MD2)  was  “The  button  labels 
accurately  represented  the  buttons’  functionality.”  Fourteen  participants  (across  conditions) 
agreed  with  the  statement.  For  MD2,  the  range  of  responses  for  the  middle  50%  of  participants 
in  the  FR  condition  was  5-6,  as  compared  to  the  range  for  the  middle  50%  of  participants  in  the 
latency  condition,  which  was  5-7.  The  statement  for  MD5  was  “It  was  easy  to  understand  what 
the  icons  used  on  the  map  displays  represented.”  Although  the  range  for  this  item  was  1-7,  the 
modal  response  was  6;  and,  the  frequency  of  the  mode  was  14.  This  indicates  that  these  14 
participants  all  agreed  with  the  statement.  Everyone  in  the  middle  50%  of  the  latency  condition 
selected  6  as  their  response.  For  the  FR  condition,  however,  the  range  for  the  middle  50%  was 
5-6.  Map  display  item  12,  which  read,  “The  symbols  that  were  used  did  not  make  sense,” 
resulted  in  14  out  of  29  participants  disagreeing  with  the  statement.  The  range  of  responses  for 
the  middle  50%  of  each  condition  was  1-2. 

Reporting  ( RPT ).  Item  4  of  the  RPT  section  garnered  strong  agreement  from  14  out  of  the  29 
participants.  This  item  read,  “Composing  a  SITREP  was  an  easy  task  to  accomplish.”  Even 
though  the  range  of  responses  for  the  middle  50%  of  the  latency  group  was  5-7,  whereas  that  for 
the  middle  50%  of  the  frame  group  was  6-7,  overall  this  suggests  agreement  with  the  statement. 
Despite  this  agreement,  participant  comments  provided  insight  into  some  of  the  problems  with 
composing  a  SITREP.  Participants  commented,  “easy,  but  took  too  long”  and  “The  difficulty 
lies  within  the  timing  of  the  reporting.” 

Unmanned  Vehicle  Control  and  Status  (UVCS).  The  statement  for  UVCS6  was,  “Requesting 
control  of  a  UV  was  easy  to  do.”  The  modal  response  for  this  item  was  seven,  indicating  strong 
agreement  with  the  statement  (across  conditions).  Participant  comments  to  clarify  the  responses 
were,  “Sometimes  it  would  stall  or  take  a  long  delay”  and  “. .  .once  steps  were  learned.” 

Teleoperation  (TELE).  Item  1 1  of  the  teleoperation  section  of  the  questionnaire  stated,  “There 
was  a  significant  time  delay  that  made  controlling  the  armed  reconnaissance  vehicle  (ARV) 
difficult.”  Fourteen  participants  disagreed  with  this  statement.  Everyone  in  the  middle  50%  of 
the  Latency  group  selected  2  as  their  response,  whereas  the  response  range  for  the  middle  50%  of 
the  Frame  group  was  2-3.  This  suggests  overall  disagreement  with  the  statement.  One 
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participant  from  the  latency  condition  commented,  “I  would  veer  off  path,  but  that  was  due  to 
speed.” 

Target  Acquisition  (TA).  There  was  only  one  item  in  which  almost  half  of  the  participants  (13 
out  of  29)  responded  in  agreement  with  one  another.  That  item  was  TA9,  which  stated,  “The 
button  labels  accurately  represented  their  intended  functions.”  The  participants  agreed  with  this 
statement.  For  one  particular  TA  item,  TA8,  there  were  multiple  modal  responses;  however,  the 
participants’  comments  provide  considerable  information.  The  item  read,  “The  aiming  reticule 
was  easy  to  use.”  The  participant  comments  were,  “should  be  a  dot  or  +,  not  an  open  space  with 
lines  on  the  outside  of  it,”  “hard  to  maneuver  crosshairs  with  UAV  and  UGV,”  “finer  tuning 
should  be  option  instead  of  aiming  next  [to],  below  target,”  “too  sensitive  because  depressing  the 
palm  grips  reduces  motor  functions  of  tester.” 

Crew  Display  and  Screens  (CDS).  Fourteen  of  29  participants  agreed  with  the  statement  for 
CDS26,  which  read,  “Color  coding  on  the  crewstation  displays  is  helpful.”  One  participant, 
however,  commented,  “not  much  color  coding  used.” 

Yoke  and  Pedal  Assembly  (YPA).  The  statement  for  Item  2  of  the  YPA  section  read,  “The 
buttons  on  the  yoke  were  in  locations  that  are  easy  to  reach.”  Fifteen  out  of  29  participants 
strongly  agreed  with  this  statement.  However,  one  participant  commented,  “The  yoke  was  far 
away  from  me  (seat  couldn’t  move  up  more).” 


II.  Selected  Comments  on  Usability  from  Participants.  (Note:  Item  numbers  do  not  match 
those  in  the  survey) 

Map  Display 


1.  I  used  the  Mapping  display  often. 

“If  I  saw  a  bad  guy  on  the  map,  it  was  hard  to  find  him  with  the  UAV.  It  was  like  I  was  looking  at  two  different 
screen  maps.” 

2.  It  was  easy  to  understand  what  the  icons  used  on  the  map  displays  represented. 

“Sometimes  I  would  confuse  the  UAV/UGV.” 

“Vehicle  icons  were  not  distinctive  enough  for  quick  recognition  w /  large  scale  incoherent  (multiple  vehicles)” 

3.  Icons  representing  different  ARVs  were  easy  to  distinguish. 

“Sometimes  I  would  confuse  the  UAV/UGV.” 

“a  little  difficult  to  tell  which  was  which  without  moving  vehicles” 

“couldn't  tell  the  difference  between  it  and  tank  “ 

4.  Geographic  orientation  of  the  map  terrain  features  makes  sense. 

“more  color  for  topography  switch  between  visual/topographical/graphic  control/road/tactical  to  filter  out 
unnecessary  input” 

5.  The  orientation  of  the  ARVs  is  represented  clearly. 

“used  compasses” 

“could  differentiate  back  and  front  of  teleoperation  by  field  of  view  icon.” 

“hard  to  tell  where  they  are  facing” 

6.  I  would  like  a  way  to  customize  the  background  colors  on  the  map. 

“It's  easier  to  learn  one  consistent  format.” 

“more  color  for  topography  switch  between  visual/topographicaFgraphic  control/road/tactical  to  filter  out 
unnecessary  input” 


48 


7.  The  map  was  my  main  source  of  situation  awareness. 

“used  compass  more  to  gauge  and  map  to  confirm  direction” 

8.  It  would  have  been  helpful  to  have  more  information  presented  on  the  map  display. 

“Topographical  info  like  elevation  from  sea  level  of  different  geographical  areas” 

“more  color  for  topography  switch  between  visual/topographical/graphic  control/road/tactical  to  filter  out 
unnecessary  input” 

9.  Overall  the  mapping  display  was  easy  to  use. 

“Use  of  a  trackball  in  a  touch  screen  interface  was  backwards,  counterproductive.  At  least  a  touch-pad  to 
coordinate  mental  functions  if  not  fully  touch  screen  integrated  map  interface” 

Reporting 

1.  I  sometimes  felt  ‘lost"  navigating  the  reporting  display  screens. 

“It  would  be  nice  to  have  larger  buttons  that  showed  the  nav  tree  better  (previous  choices  stay)  “ 

Unmanned  Vehicle  Control  &  Status 

1.  The  UV  control  display  was  easy  to  use. 

“should  have  a  joystick  for  UAV” 

Teleoperation 

1.  The  yoke  made  it  difficult  to  control  the  ARVs. 

“too  sensitive  for  turret”  (Latency  group) 

2.  The  yoke  was  too  sensitive  to  control  ARV  direction. 

“a  little  bit  too  sensitive”  (FR  group) 

3.  The  video  update  rate  was  sufficient  to  teleoperate  the  ARV  at  high  speed. 

“could  not  go  faster  than  20  without  losing  control”  (Latency  group) 

4.  There  was  a  significant  time  delay  that  made  controlling  the  ARV  difficult. 

“I  would  veer  off  path,  but  that  was  due  to  speed.”  (Latency  group) 

Target  Acquisition 

1.  The  TA  display  should  contain  additional  information. 

“better  direction  indicates  when  switching  between  ARVs” 

“Target  info” 

2.  The  aiming  reticule  was  easy  to  use. 

“should  be  a  dot  or  +,  not  an  open  space  with  lines  on  the  outside  of  it” 

“hard  to  maneuver  crosshairs  with  UAV  and  UGV”  (FR  group) 

“Finer  tuning  should  be  option  instead  of  aiming  next,  below  target” 

“too  sensitive  because  depressing  the  palm  grips  reduces  motor  functions  of  tester”  (Latency  group) 
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General  Usability 


I.  Crewstation  Display  and  Screens. 

1.  The  touchscreen  buttons  were  large  enough  to  use  effectively. 

“The  mission  status  screen  was  too  small.” 

“Did  not  respond  well  to  a  quick  change” 

“These  buttons  were  hard  to  use  effectively” 

“not  when  it  came  to  scrolling  lists” 

“except  scroll  button” 

2.  The  point  at  which  my  finger  touches  is  not  identical  to  the  point  indicated  by  the  touchscreen 
(there  are  parallax  problems). 

“only  with  the  scroll  menu”  (Latency  group) 

“Only  on  the  scrollbars”  (Latency  group) 

3.  I  preferred  using  the  touchscreen  to  the  trackball  for  interacting  with  the  displays. 

“the  track  ball  was  more  precise  and  easy,  but  touchscreen  would  be  better  if  worked  better” 

“trackball  is  unwieldy  when  trying  to  act  and  react  quickly” 

4.  The  crewstation  presents  system  messages  in  a  consistent  format  and  location. 

[formation  was  not  centralized” 

5.  Overall,  I  think  the  crewstation  was  easy  to  work  with. 

“except  the  map  w/  trackball  that  was  'backwards'  and  frustrating” 

II.  Yoke  and  Pedal  Assembly. 

1.  The  buttons  on  the  yoke  were  easy  to  use. 

;ccpt  palm  grips  for  turret  control” 

2.  The  yoke  was  too  sensitive  to  over-steer. 

“somewhat  sensitive”  (FR  group) 

“at  high  speeds,  yes”  (Latency  group) 

3.  It  was  easy  to  slew  to  the  target  using  the  yoke. 

“a  little  difficult  to  place  cursor  dead  on”  (Latency  group) 

4.  It  was  easy  to  maintain  crosshairs  on  the  target  with  the  yoke. 

“need  more  fine  tuning  capabilities”  (FR  group) 

“too  sensitive  because  my  hands  had  to  squeeze  the  palm  grips;  this  reduced  my  overall  hand  coordination” 
(Latency  group) 


50 


Glossary  of  Acronyms 


ARV 

armed  reconnaissance  vehicle 

CDS 

crew  display  and  screens 

ECATT-MR 

Embedded  Combined  Arms  Team  training  and  mission  rehearsal 

FCS 

Future  Combat  System 

HRI 

human-robot  interaction 

ID 

identification 

MCS 

Mounted  Combat  System 

MD 

map  display 

OCS 

operator  control  station 

OneSAF 

semi-automated  forces 

RDECOM 

Research,  Engineering,  and  Development  Command 

RPT 

reporting 

SA 

situational  awareness 

TA 

target  acquisition 

TS 

total  severity 

UAV 

unmanned  aerial  vehicle 

UCD 

unmanned  combat  demonstration 

UGV 

unmanned  ground  vehicle 

UV 

unmanned  vehicle 

YPA 

yoke  and  pedal  assembly 
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THE  UNIV  OF  TEXAS  AT  AUSTIN 
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AUSTIN  TX  78759-5316 

1  DIRECTOR 
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BLDG  4011  RM  217 
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1  ARMY  RSCH  LABORATORY  -  HRED 
ATTN  AMSRD  ARL  HR  MG  R  SPINE 
BUILDING  333 

PICATINNY  ARSENAL  NJ  07806-5000 

1  ARMY  RSCH  LABORATORY  -  HRED 
ATTN  AMSRD  ARL  HR  MH  C  BURNS 
BLDG  1002  ROOM  117 
1ST  CAVALRY  REGIMENT  RD 
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1  ARMY  RSCH  LABORATORY  -  HRED 
AVNC  FIELD  ELEMENT 
ATTN  AMSRD  ARL  HR  MJ  D  DURBIN 
BLDG  4506  (DCD)  RM  107 
FT  RUCKER  AL  36362-5000 

1  ARMY  RSCH  LABORATORY  -  HRED 

ATTN  AMSRD  ARL  HR  MK  MR  J  REINHART 
10125  KINGMAN  RD 
FT  BELVOIR  VA  22060-5828 

1  ARMY  RSCH  LABORATORY  -  HRED 
ATTN  AMSRD  ARL  HR  MV  HQ  USAOTC 
S  MIDDLEBROOKS 
91012  STATION  AVE  ROOM  111 
FT  HOOD  TX  76544-5073 

1  ARMY  RSCH  LABORATORY  -  HRED 
ATTN  AMSRD  ARL  HR  MY  M  BARNES 
2520  HEALY  AVE  STE  1 172  BLDG  51005 
FT  HUACHUCA  AZ  85613-7069 
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1  ARMY  RSCH  LABORATORY  -  HRED 

ATTN  AMSRD  ARL  HR  MP  D  UNGV ARSKY 
BATTLE  CMD  BATTLE  LAB 
415  SHERMAN  AVE  UNIT  3 
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1  ARMY  RSCH  LABORATORY  -  HRED 

ATTN  AMSRD  ARL  HR  MJK  MS  D  BARNETTE 
JFCOM  JOINT  EXPERIMENTATION  J9 
JOINT  FUTURES  LAB 
115  LAKEVIEW  PARKWAY  SUITE  B 
SUFFOLK  VA  23435 

1  ARMY  RSCH  LABORATORY  -  HRED 

ATTN  AMSRD  ARL  HR  MQ  M  R  FLETCHER 
US  ARMY  SBCCOM  NATICK  SOLDIER  CTR 
AMSRD  NSC  SSE  BLDG  3  RM  341 
NATICK  MA  01760-5020 

10  ARMY  RSCH  LABORATORY  -  HRED 
ATTN  AMSRD  ARL  HR  MT  DR  J  CHEN 
12350  RESEARCH  PARKWAY 
ORLANDO  FL  32826-3276 

1  ARMY  RSCH  LABORATORY  -  HRED 

ATTN  AMSRD  ARL  HR  MS  MR  C  MANASCO 
SIGNAL  TOWERS  RM  3  03 A 
FORT  GORDON  GA  30905-5233 

1  ARMY  RSCH  LABORATORY  -  HRED 

ATTN  AMSRD  ARL  HR  MU  M  SINGAPORE 
6501  E  11  MILE  RD  MAIL  STOP  284 
BLDG  200A  2ND  FL  RM  2104 
WARREN  MI  48397-5000 

1  ARMY  RSCH  LABORATORY  -  HRED 

ATTN  AMSRD  ARL  HR  MF  MR  C  HERNANDEZ 
BLDG  3040  RM  220 
FORT  SILL  OK  73503-5600 

1  ARMY  RSCH  LABORATORY  -  HRED 
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FT  BENNING  GA  31905-5400 
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FORT  BRAGG  NC  28310-5000 
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1  DIRECTOR 
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1  US  ARMY  TACOM  ARDEC 
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BLDG  151 

PICATINNY  ARSENAL  NJ  07806-5000 

1  US  ARMY  TACOM  ARDEC 

SMALL  CALIBER  WEAPON  SYSTEMS 
ATTN  AMSRD  AAR  AEW  M(D)  MR  TORRES 
PICATINNY  ARSENAL  NJ  07806-5000 

1  US  ARMY  RDECOM- ARDEC 

LWDMW  STO  MANAGER 
ATTN  AMSRD  AAR  AEW  F  (D)  J  SANTIAGO 
BLDG  61  NORTH 

PICATINNY  ARSENAL  NJ  07806-5000 

1  US  ARMY  RDECOM- ARDEC 

LWDMW  STO  MANAGER 
ATTN  AMSRD  AAR  QEM  C  M  VOIT 
BLDG  62  SOUTH 

PICATINNY  ARSENAL  NJ  07806-5000 

1  US  MILITARY  ACADEMY 
OFFICIAL  MAIL  DIST  CTR 
ATTN  MADNBSL  CPTJASLOAN 
646  SWIFT  ROAD 
WEST  POINT  NY  10996-1905 

ABERDEEN  PROVING  GROUND 

1  DIRECTOR 

US  ARMY  RSCH  LABORATORY 
ATTN  AMSRD  ARL  Cl  OK  (TECH  LIB) 

BLDG  4600 
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